forked from wtsi-npg/npg_qc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Changes
1621 lines (1376 loc) · 77.5 KB
/
Changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
LIST OF CHANGES FOR NPG-QC PACKAGE
release 66.1.1
- composite genotype check script (call_gtck_composite_rpt.pl) search
for run folders disabled - only use iRODS cram files
release 66.1
- autoqc results loader to check for potential composition digest clashes
- amend upstream_tags check to find previous runs which have reached
status "analysis complete" (instead of "run complete")
release 66.0
- GUI for utility QC (inactive)
- add new bcfstats qc check + viewer change
- skip incomplete tag zero results in genotype_call POSTing to LIMs
- minor changes to genotype qc check to support checking against externally
supplied genotypes instead if they exist
- handle target samtools stats file - parsing, archiving and viewing.
- queries to qc_store should always have support for tracking db look-up
- methods for loading autoqc results moved from npg_qc::autoqc::collection
to npg_qc::autoqc::qc_store
- little-used method load_run of npg_qc::autoqc::qc_store class, which did
not support propagation of tracking db handle, is removed
- unused public methods in npg_qc::autoqc namespace removed
- mqc outcomes API - a new method, get_library_outcomes, for retrieving
library qc outcomes as boolean values
- deciding whether the entity is subject to manual QC:
remove special provisions for GCLP;
enforce that libraries and lanes cannot be qc-ed on the same page
- enable saving qc outcomes for multi-component compositions
- SeqQC GUI changes for multi-component compositions:
enable display of autoqc results;
enable manual QC
- changes to allow for SeqQC display of results corresponding to multi-
component compositions
- Illumina QC data loader restricted to loading cluster density data, its
data pre-loading functionality dropped
- amend spatial_filter check to take a directory name to search instead
of an explicit list of input files (that approach became unworkable
with highly plexed lanes)
- add result_file_path attribute to file-based result objects
- samtools stats file parser
- qX_yield and gc_fraction autoqc checks to use samtools stats files
instead of fastqcheck file
- SeqQC:
links for latest ivc and analysis removed since the target files are
no longer produced;
link for full rna seqc analysis follows info in the result object
- remove can_run restriction of verify_bam_id check for (cD|R)NA libraries
release 65.4.1
- store value of output_dir from the RNA-SeQC check into the database
release 65.4
- qX_yield autoqc check extension: capture and save yields for q30 and q40
release 65.3
- fixed bug in cluster_density calculation where no TileMetrics interop
file exists
release 65.2
- add new metric mitochondrial genes content as mt_pct_tpm to RNA-SeQC check
- can_run returns false for pulldown_metrics check for tag 0
- genotype check does not require the existence of irods: input_files
release 65.1.1
- explicitly use pre v6 bwa for rna_seqc rRNA alignment as input not name sorted
(tolerated in old version but not bwa0_6)
release 65.1
- bwa to bwa0_6 for rna_seqc rRNA alignment
release 65.0
- adapter autoqc check - discard code for fastq input
- add genotype_call_results_reporter script to report genotype_call
results to the LIMs
- tag_metics matches_pf_percent only stored to three decimals places now
so trim in viewer
- modified spatial filter qc check to read all run_lane filter.stats files
- changed naming of spatial_filter stats files to <run>_<lane>*.spatial_filter.stats
- upstream_tags qc check - replaced bamindexdecoder with bambi decode
- dropped ability for qc checks to sub-sample input fastq
instead run the checks directly off the cached fastq files
- NovaSeq changes, new InterOp file format and per lane read counts now 64-bit unsigned ints
- make travis happy as running apt-get update by default was disabled.
- NovaSeq run info file size is around 95KB, too large for the text column type.
Column type changed to mediumtext.
- upstream_tags autoqc check: standardise access to the tracking database
- 'execute' method of the parent autoqc check to error if input files do not exist
- dependency on tracking XML feeds removed
- bwa0_6 (newer bwa) expalicitly requested where bwa aligner is used
- all C source code moved to https://github.com/wtsi-npg/npg_qc_utils, executables are
built by Conda https://github.com/wtsi-npg/npg_conda/tree/devel/recipes/npg_qc_utils/65.0;
Build.PL changed accordingly
release 64.6.3
- update parsing of spatial filter stats in spatial_filter QC check
release 64.6.2
- QC viewer changes for genotype call
release 64.6.1
- tweak to GbS library type check in genotype call as arrived as GBS (now case-insensitive).
- Exit rna seqc gracefully if bam has no reads
- more stringent ref check now required for Pf3D7_v3.fa/hs37d5__Pf3D7_v3.fa
in genotype check
release 64.6
- add new metric glogin_pct_tpm to RNA-SeQC check
- DBIx schema loader script: alter table to add new globin metric column
- add functionality to pass some of the files necessary to run RNA-SeQC
as CLI options for program qc for more flexibility for stand alone run
- add new role for rna_seqc to implement ability to extract values from
column other_metrics to be stored in ml warehouse database.
- add tag_hops_power and tag_hops_percent columns to tag metrics
- remove run_timeline component on the Illumina QC loader
- stop generating DBIx class for run_timeline db table
- remove unused methods of the run_graph Clearpress model
- add new role for rna_seqc to implement ability to extract values from
column other_metrics to be stored in ml warehouse database
- changes for GbS genotype calling check
- tag_sniff
handle separators in BC tags for dual-index runs
allow dual-index runs to be revcomped independantly
when tag has a separator -t option operates on subtags
allowed to ignore an index when the BC tag contains a separator
release 64.5
- qc outcomes model: code generalisation to simplify addition of
further qc types
- uqc outcomes - saving and updating enabled
- switch to node 6.12.2 in travis
- stop generating DBIx classes for unused tables
- tiny tweaks to rna and verify bam id help links
- ensure, where practical, that saving and retrieving qc outcomes
works for multi-component compositions
- add functionality to allow more columns to be added to table dynamically
- add data for new extra column for sample_name in summary table
- treat all dynamically added methods in the same way
release 64.4.1
- fix bam_flagstats check library_size bug
release 64.4
- utility QC feature: db tables, DBIx classes, retrieval
- Add STAR aligner bams as a valid RNA alignments
- update bam_flagstats to be able run when no markdups metrics file exists
release 64.3.1
- error in a statement for data update is fixed
release 64.3
- DBIx schema loader script: automatically add npg_qc::Schema::Composition
- drop unique consctraint for (id_run, position and tag_index) columns;
- make a foreign key to seq_composition table not nullable and set
a unique constraint on the foreign key column in entity tables;
- do not use id_run, position and tag index column values from
entity tables, compute rpt lists from compositions;
- do not list explisitly id_run, position and tag_index in return
datastructes for both get and save outcomes methods
- a method to find a composition without creating one
release 64.2
- Cannot have default Moose constructor in derived DBIx classes - fix for
custom MqcOutcomeEnt resultset
- DBIx achema loader script: automatically add npg_qc::Schema::Composition
role when generating result classes for autoqc tables
- DBIx classes for autoqc db tables:
remove 'create_component' factory method which was only needed
for back-filling composition foreign key;
inherit 'composition' attribute from a newly created
npg_qc::Schema::Composition role;
document 'composition' attribute
- composition-enabled manual qc:
db tables linked to the seq_composition table;
search proceeds via releated composition tables;
when mqc outcomes are saved, a composition is created if it does not exist;
'composition' attribute is added to mqc entity result classes;
- upgrade bower to 1.8.2 in travis
- stop building on Travis under Perl 5.16,
see https://rt.cpan.org/Public/Bug/Display.html?id=123310
release 64.1
- SeqQC viewer - function to draw readonly representation of the
utility flag
- autoqc checks parect object - refactore to use a new factory for
rpt list to composition transformation
- upstream_tags modified so it can work with the rpt_list argument
- constraints in autoqc db tables are reset to allow for saving
results for multi-component compositions
- tag_sniff modified to load the whole tag table upfront
revcomp matches now output in reverse video
release 64.0
- composition-enabled autoqc objects storage
schema changes
DBIx binding updates
script for back-filling foreign keys to composition table
- bug fix on search_autoqc method
- db composition creation via a factory method
- db autoqc loader - unused 'update' option removed
- db autoqc loader - will retry a transaction that failed because
a db lock could not be aquired
- saving a component with subset attribute value 'all' gives an error
- npg_qc::autoqc::results::collection
remove unused functionality;
ensure existing methods work with objects that do not have id_run,
position, etc
- remove superfluous method call from a template
- db query for tag_index NULL or NOT NULL: undef as a string
might not give problems now, but will when all db records are
linked to compositions
- remove unused qc_chema property from TransferObjectFactory
- set explicit dist to precise in travis yml
- simplify and optmise pool detection in the viewer
- remove links to Clarity LIMs, generalise code for linking to LIMs
- deleted VC (view-controller) part of the Clearpress npg_qc web server
- index unq_seq_compos_rp for seq_component becomes unique
release 63.1
- label fix for rna_seqc in SeqQC viewer
- MySQL does not save correctly integers to float columns;
change column definitions for rna_seqc
release 63.0
- update script for deleting autoqc database results so that it
can handle composition-based tables
- remove attribute qc_report_dir from check object: the output directory
is created by default using the sample's filename root
- npg_qc::autoqc::qc_store - load rna_seqc results into new rna_seqc
table in npg_qc database.
- SeqQC:
- added template for rna_seqc check with selected metrics shown
in summary
- include a link to original RNA-SeQC report in check's template
- Build will fail if no node, npm or bower are available in path
- stop using multiple versions of samtools
- do not install NPGQC web app
- increase HTTP timeout when reporting manual qc outcomes to LIMs
release 62.9
- fix for GD image generation test in response to a change in constructor
behaviour in GD.pm
- compute expected path in tests in a way that takes into account
substitutions
- genotype and verify_bam_id checks:
speed up pipeline job submission for deeply plexed lanes by
moving calls that involde access to reference repository out of the
can_run method
- composition-based autoqc results db search via DBIx: allow for
a search query that contains fields from the main table
release 62.8
- tag_sniff modified clip and revcomp options to handle split tags
- Sort results for merged genotypes as well - to avoid out of order failure.
- specify an appropriate "accept" header when sending manual qc
outcomes to LIMs (header was not set resulting in 406 responce
code)
release 62.7
- current staging directories do not have autoqc results in
bustard_path; stop looking there
- ClearPress Perl library is pinned to v. 473.0.5
- upstream_tags autoqc check: stop falling back on files in IRODs
release 62.6
- make building the iRODS handle of npg_qc::autoqc::checks::genotype
lazy to avoid calling baton until required at runtime
- translation from a database composition representation to the
npg_tracking::glossary::composition type object
- db query for compisition-based tables should include a condition
on tag_index if only lanes or plexes are being retrieved
- Fix in npg_qc::utils::bam_genotype for rare out of order mpileup failures
- Travis CI build: when building dependencies, us the same
branch as the one the pull request is made to.
- back to testing with mysql 5.7.13 in travis
- drop node 0.12 from node travis matrix
- add perl 5.22-shrplib to travis matrix
- to reduce uncertainty in ranking close results, increase precision
used for ranking contamination results
- call_gtck_composite_rpt.pl - supply rpt list to genotype check
- Code and tests changes to reduce number of warnings under Perl 5.22.2
- sequence mismatch - do not return PDL special values, treat NaN as
a bad value and return undefined instead
release 62.5
- a very concise replacement for NPG_QC pages
- no templates rna_seqc results - we should not try rendering
release 62.4
- autoqc loader does not try loading a result for which there is
no coresponding DBIx binding
release 62.3
- make bam_flagstats check runnable under the qc script
- remove autoqc loader backwards compatibility with analysis code
that does not generate related results for the bam_flagstats results
- Arguments given to the qc script are passed through to the relevant
qc check object.
- QC script does not try to infer the location of input files.
- QC script accepts the rpt_list argument.
- tag_index attribute of the check object cannot be assigned to undef.
- added Broad Institute's RNA-SeQC to autoqc QC checks
- SeqQC - genotype view fixed (escaping)
release 62.1
- Bug fix in build action for cgi files.
release 62.0
- Towards composition-based autoqc:
- npg_qc::autoqc::results::result - Composition-aware,
id_run, position and path attributes become optional.
- npg_qc::autoqc::qc_store - Use DBIx syntax for quering on tag_index
(future compatibility with planned table schema changes), db load
compositions of size 1 only.
- npg_qc::autoqc::db_loader - Result classes support composition,
but most tables have not been converted yet. Do not generate
composition if no support in the result table.
- npg_qc::Schema::ResultSet - Enchancement for a search for tables that
do not yet support compositions: convert undefined values
for columns under a unique constraint to database defaults.
- SeqQC - outdated template for split_stats removed
- genotype role refactored to remove hardcoded uri encoding
- remove contamination check that is not run any longer,
result object remains
- to keep similar code together and to avoid listing explicitly
an increasing number of non-runnable checks, move checks_list()
method to the collection object
- genotype check, croak if error closing samtools pileup command
- norm_fit
- modify peak finding to allow for peaks in first/last bin
- added monotonity condition when estimating stdev
- drop stdev when outputting filtered modes
release 61.2
- SeqQC:
- Reintroduce collapser
- using grunt/npm to automate js testing
- VerifyBAMId changed condition MIN_AVG_DEPTH from > 4 to > 2 for fail
release 61.1
- SeqQC
- Using --force-latest for bower install deps during Build
- improve test robustness (larger page sample for mech to check)
release 61.0
- using test matrix with node 4.4.2 and 0.12.9 for travis
- help page for manual qc
- redesign of both client and server side for generic qc outcome updates
- updates to already stored outcomes, including final outcomes,
are not an error, will be skipped
- require that all library outcomes are marked explicitly before the
final sequencing outcome is saved
- custom class for authentication in tests - works with JSON POST requests
- mqc widgets not shown if no lims data available
- title for pages with data for one id run show run status
- SeqQC
- bugfix only build request for qc_outcomes if rptkeys in page
- upgrading jquery to 2.2.3 from 2.1.4
- upgrading requirejs to 2.2.0 from 2.1.20
- upgrading bcviz to 1.3.2 from 1.3.1
- upgrading table-export to 1.5.0 from 1.2.2
- upgrading qunit to 1.23.0 from 1.20.0
- add unique db constraint for short descriptions in qc outcome
dictionaries; the constraints should have been set when the tables
were created
release 60.0
- JSON service (retrieval) for qc outcomes
- update link to picard in POD
- SeqQC:
- remove use of autocrud plugin - it's never been used
- simplify the authorisation and it make more secure (do not fetch
user credentials from the url), move the code to a new User model
- in tests extend the User model to allow for getting user credentials
from the url
- avoid warnings in tests by setting the default for the username to
an empty string
- if the user is logged in and is authorised to do manual qc, display
a visual cue on the top of the page
- tag metrics column moved next to the column with tag index and tag sequence
- show deplexing percent for individual tags in tag metrics column
- retrieve qc outcomes from npg_qc_viewer qcoutcomes JSON service for
initial page rendering
release 59.8
- give LIMs time to process posting qc outcomes as individual events
- SeqQC
- upgrading bcviz to 1.3.1 from 1.3.0 (individual minified files)
release 59.7
- SeqQC
- bower configuration files in npg_qc/npg_qc_viewer
- moving client test files to npg_qc/npg_qc_viewer/t/client
- fix regression, missing require for table-export
release 59.6
- SeqQC
- generating bcviz plots only when they will become visible and remove
them when they will not be visible
- set max number of plexes for manual QC as 400
release 59.5
- autoqc results db loader: an option to force a re-build of related objects
- SeqQC
- fix warning about undef value in the adapter template
- fix closing div tag in verify bam id template
- upgrading jquery to 2.1.4 from 2.0.3
- upgrading requirejs to 2.1.20 from 2.1.8
- upgrading qunit to 1.20.0 from 1.15.0
- upgrading bcviz to 1.3.0 from 1.2.1
- travis ci testing perl modules for npg_qc and npg_qc_viewer
- require DBD::SQLite@1.48 to run tests
release 59.4
- saving manual qc values for libraries - allow for an undefined or empty
list of tag indices
- tags in lanes from GCLP runs are not subject to library qc
- composition-based autoqc results - disable version checking between the version
of the module that serialized the object and the version of the same module that
is performing de-serialization
- script to import manual qc values from the old warehouse RT#493757
- update_outcome method of mqc entities renamed to update_nonfinal_outcome
- a new update_outcome method allows for update of final outcomes
- method to toggle stored final manual qc outcomes
- mqc reporter - reporting might cause a loss of link between the product and
flowcell data in the ml warehouse, so prepare all data, then report
- SeqQC
- New functionality to allow export summary table to CSV
release 59.3
- transparent search query for autoqc objects irrespectively of the details of
object's database implementation (in-table identifies or composition)
- custom parent class for resultset objects
- calling ->new() on resultset object to create a new result replaced with
calling more intuitive ->new_result()
- upgrade genotype check to use samtools1 and bcftools1
- SeqQC
- New dictionary table for plex level library MQC. (Issue 238)
release 59.2
- library-level manual QC for individual plexes.
release 59.1
- 'write2file' method of autoqc result objects removed since it duplicated
the 'store' method
- alternative base class npg_qc::autoqc::results::base for autoqc results objects,
able to represent results that are derived from merged files and described by
a composition of multiple components
- autoqc result objects for samtools stats file, sam/bam headers and digests
derived from the new base class
- bam_flagstats result extension to produce samtools_stats and sequence_summary
results (related objects) either at run time or later
- database tables for storing component and composition objects
- autoqc db loader extension
- to load data represented by classes derived from the new base class
- to build related objects in memory and to load them to the database
bypassing serialization to a file - a way to save cram headers and stats
file where the analysis pipeline did not run the current code that
produces samtools_stats and sequence_summary objects as individual files
- ability to mark records as current
- reporter for manual QC results retrieves LIMs info from ml warehouse;
it skips GCLP runs RT#480489.
- tag_sniff new option to truncate tag sequences
- tag_sniff checks common tag sequences against tag sets in the warehouse
- SeqQC
- add the total yield in library view RT#488973
- summary table: display supplier sample name along the library barcode RT#488964,
remove separate sample column
- fix missing composite genotype check visualisation broken when cleaning
template for ml warehouse
- fix missing space between forward and reverse when reporting percent gc
fraction
- genotype check data extraction scripts - updated scripts to fail more readily
and informatively when an iRODS error occurs
release 59.0
- SeqQC
- remove ajax proxy controller
- stop 00 tests accessing live db credentials and databases
- using mlwarehouse for dwh information displayed in SeqQC viewer
- removing tests related to template compilation
- providing links to lims for pool/library/sample
- autoqc check and autoqc objects: remove unused 'sequence_type' attribute
- autoqc check and result objects - use existing roles for id_run and
position definitions
- result object - to remove dependency on id run, position and tag index,
allow individual results to overwrite the root of the output file name
- refactor autoqc check to use file_type attribute name instead of
input_file_ext
- tags_reporters check - remove hardcoded solexa path
- genotype and verify_bam_id_checks - use standard way of reporting run-time
problems
- bam_flagstats
- one call to compute all metrics
- method for finding stats files
- if id_run not given, derive the root of the output file name from
the input sequence file name
release 58.9
- move method for file name creation from SeqQC to autoqc so that it's
more accessible
- SeqQC
- upstream_tags template - do not report potentially confusing tag index value when
there is no id_run in the high-scoring tags table
- remove misleading bam_flagstats check info
release 58.8
- SeqQC
- display insert size normal fit confidence and number of modes in the
summary table
- do not show adaptor plots for low contamination
- provenance in lane title not in every check
- prepend all page titles with application name and version
- different page title colour if running in non-live environment
- use updated LIMs server url
- link to NPG tracking web server running on the same host as this app
- showing reference species and version below lane titles
- Using container configuration for travis
release 58.7
- bam_flagstats autoqc result object:
new field, subset, added to be used instead of human_split
human_split is retained for now to be compatible with existing data and code
- autoqc loader will only load fields that correspond to database columns
release 58.6
- Using "jquery-unveil" plugin for heatmaps
- Removing data from DOM after passing them to local variables
- Nulling variables after using them for bcviz to free memory
- Moved simplified file name generation to this package
- Removing gc bias and qX Yield checks when looking at non-run views
- Removing collapser
- Using latest release of bcviz (1.2.1)
release 58.5
- stop using depricated Class::MOP::load_class method
- use namespace::autoclean instead of "no Moose" in Moose objects
- using make_immutable in Moose objects
release 58.4
- Creating a cache in SeqStore model to reduce file system access.
Improving query - using keys for tables.
- Moving FileFinder functionality into SeqQC from seq_commons.
- Cache of catalyst actions and uri in template.
- Templates are configured to be checked for changes every 10 hours.
- Connections to database for fastqcheck are passed to model from
template.
- Test modules catch/test for warnings for unitilized ids and for
cases where it was not possible to locate a folder.
release 58.3.1
- Updating Readme in npg_qc_viewer for changes in deploy procedure.
- Fixing qXYield total for run and sample, now it calculates from actual values
in the table, does not recalculate from collection.
- genotype check: reduce min_sample_call_rate parameter for genotype match from 95% to 75%, and
min_common_snps from 21 to 20. This will allow (Fluidigm) qc genotypes with 20 calls to be
identified as possible duplicates when using the standard set of 26 QC SNPS (W30467)
release 58.3
- default autoqc result object creation - do not set tag index to undef to
avoid this field being set when serializing to json with newer Moose
release 58.2
- convert local path to nfs for linked tools when building fastq_summ
release 58.1
- compile fastq_summ with samtools-1.2 and htslib-1.2.1
- Removing global functions from manual_qc.js
- Logic for preliminary outcomes for MQC
- New undecided outcome for MQC
- update scripts for genotype check data preparation
- to allow specification of iRODS zone with an environment variable
- perlcritic issues and version strings
- gt_pack recognises version 02 headers in append mode
- DBIx binding generation - keep column name case when generating accessors
release 58.0
- skip running legacy gcbias if window_depth not available
- compile fastq_summ executable during the npg_qc build, skip
compilation if samtools is not on the path
- C code clean-up to remove warnings in compilation
- SeqQC daemon to set CATALYST_HOME relative to bin so that
the production user does not have to have this definition
- SeqQC build optionally fetches javascript dependencies
- added scripts to handle generation of data for autoqc genotype checks
release 57.12
- Removing MQC logic from templates and moving it to JS
- Adding a controller to provide MQC status for a run using REST
- Adding a role to provide functionality for 401 responses
- Validation for DWH->MQC outcome integrity during mqc for logged mqc-ing user
- Fixing id-username data in test data to fix patches in controllers
which validate username
release 57.11
- take into account old-style tag metrics results when deciding
whether the lane has plexes
- be compatible with latest changes to the db_connect role
- ensure that tests do not access user's home directory
release 57.10
- total_reads now returns 0 when num_total_reads is 0 rather than undef
- add support for metrics files produced by bamstreamingmarkduplicates
- npg_qc build for web server does not compile executables
- functionality of the util package in SeqQC merged into the rpt_key autoqc role,
the package removed
- api::error package moved to Util::Error
- using node-qunit-phantomjs to run headless tests
- tests for basic functionality for mqc Javascript code
- adding configuration for blanket
- updating readme for changes in testing procedure
release 57.9
- autoqc results collection class - add a method for inferring whether
a lane has plexes from results themselves
- SeqQC - steps towards making display of autoqc results and manual qc process
independent of availability of warehouse data:
- use the new collection method to decide whether to display a link to
plex results (previously warehouse data were used for this)
- if plex results are available, always display a link to them
- SeqQC - remove vestigial code for batch id retrieval
- SeqQC build: remove --linkable-bcviz-path option since dependency on bcviz
is now managed by bower
- SeqQC: Ajax controller removed (was used as a proxy for now discontinued
requests to LIMs web server)
- following restructuring of bcviz, amend statements for import of bcviz libraries
- changes to allow genotype checks to be run on cram files
- list of human references (with chr.naming convention) used by genotype check
moved to file in the genotype repository
- GRCh38 human references added to the list
- gt_utils (gt_pack and find_gt_match C programs) added to Build install
release 57.8.1
- add temporary file name option to bamtofastq command
release 57.8
- Using bower to manage javascript component dependencies.
- Using bcviz v1.1.0 which now supports bower.
- use Biobambam bamtofastq in place of Picard utility in adapter qc check
release 57.7
- removed redundant methods from NpgDB model
- New javascript functionality to process mqc. MQC outcomes are changed using AJAX
calls to the controller directly.
- mqc controller now provides with the functionality for mqc through asynchronous calls.
- mqc controller does not saves a log in the tracking database when mqc outcomes are
updated.
release 57.6
- Added mqc_outcome_reporter
- Added a new column to the mqc entity and historic table to keep separate track of who
updates the record through the web page and who was the last user to modify the row.
Updates to tests to deal with the new schema.
- MqcOutcomeEnt validates outcomes are final before updating them as reported.
release 57.5
- amend location of composite genotype data (composite_results_loc) in npg_qc_viewer.conf
release 57.4
- in order to be able to retrieve values of foreign keys without
fetching the row in the parent table, DBIx classes generated with
an option to drop id_ at the start of the relation name
- InflateColumn::Serializer component added selectively to autoqc classes
rather than to all classes
- mqc models (DBIx) functionality extended to create a method for
updating/creating outcomes
- column name fixed in the mqc_outcome_hist table
- SeqQC viewer: top horizontal menu removed, Help link added to the menu on the right
- SeqQC viewer: NPG links point to correct production/dev servers
- SeqQC viewer: superfluous javascript code removed
- SeqQC viewer: checks/runs page removed
- SeqQC viewer: links to individual entities on the right removed
- SeqQC viewer: if results for one run only are displayed and the request was not explicitly
from staging or path, a link back to the run SeqQC page is displayed on the right
release 57.3
- SeqQC: removed pages that are not used any more (studies, samples, libraries,
people, weekly reports)
release 57.2
- updated the way bcviz functions are called from SeqQC
- Added tables for manual qc.
release 57.1
- d3 bcviz rendering of adapter, insert size and sequence mismatch plots
- page rearrangement to display ref match and old contamination results side-by-side removed
- SeqQC templates, client-side scripts and config files installed to a directory
under the install_base path; if a bcviz path is given, a link to it is created,
old bcviz link is overwitten in this case
- removed generation of google chart api urls
release 57.0
- incorporated SeqQC viewer
release 56.16
- changed default data source for call_gtck_composite_rpt.pl to combined Sequenom/Fluidigm results
release 56.15
- if norm_fit test produces no output, e.g. if no peaks after filtering, simply add a comment
release 56.14
- updates to norm_fit code
more informative log messages
added checks for zero peaks
do not run norm_fit test if there is not enough variation in the insert size
release 56.13
- correct size of genotype_data_set column of genotype table
release 56.12
- insert_size.norm_fit_confidence from integer to float
release 56.11
- changes to genoype check and utils to use combined Sequenom
and Fluidigm results for the 26 QC SNPS (W30467)
- fixed RE when parsing norm_fit output
- fixed a bug and cleaned up norm_fit code
release 56.10
- support for search for autoqc objects without tag_index attrubute
- Build.PL extended to compile and deploy norm_fit executable
- only run verify_bam_id if single sample expected (helps cope with
single plex in pool)
release 56.9
- database name-agnostic database dump file
release 56.8
- increase DB field sizes for genotype qc check results
release 56.7
- only insert cluster densities which exist
release 56.6
- use length of index read from tag0 bam file rather than length of tag in
decode metrics file to derive barcode file, added new tests and test data
- do not croak if a Bustard_Summary file does not exist (as is the case
for HX instruments)
- made interop parsing code more generic
release 56.5
- phix bam_flagstat to have default serialisation name including phix
release 56.4
- get cluster densities from InterOp/TileMetricsOut.bin
- added test data for tracking database
release 56.3
- added check for alignments, reference and library_type to can_run()
release 56.2
- relax regexp for class name in result role to allow git or svn version numbering
release 56.1
- git-based module and script versions
- RSC keywords removed
release 56.0
- use test data that are publicly available
- getting autoqc module versions made to work with non-numeric versions
- redundant Gerald summary loader code removed
- unused test data removed
- in test runfolders that are used GERALD directories renamed to PB_cal
since gerald_path is going to be removed from npg_tracking::illumina::run::folder
release 55.10
- remove default for verify_bam_id.pass
release 55.9
- minor bug fix for verify_bam_id
release 55.8
- remove Fey modules from npg_qc::autoqc::db namespace, t/60-autoqc-db*.t tests
and test data not used elsewhere
- qc_db_delete_autoqc script:
updated help message
fix for a bug that resulted in no data deleted unless a list of checks to
delete was given explicitly
release 55.7
- tag sniff usage message
- minor code changes to a couple of checks
- test/test data for verify_bam_id
- preserve case when generating DBIx binding since verify_bam_id
has mixed case column names
release 55.6
- exclude verify_bam_id (not run by the pipeline yet) from checking a set
of archived autoqc results for completeness
release 55.5
- patch for bin/call_gtck_composite_rpt.pl
release 55.4
- patch for npg_qc::autoqc::checks::upstream_tags - extra imports required
release 55.3
- 'check' option for the script deleting autoqc results from a database
- unused Fey-based autoqc loader module removed
- contamination autoqc check table: drop not-null constrain for fields with
serialized data and replace empty strings there with nulls -
InflateColumn::Serializer DBIx plugin gived error trying to convert
empty string to JSON
- DBIx binding generation extension - DBIx result classes for tables
with autoqc results consume autoqc roles
- autoqc data retrieval via the DBIx db binding
release 55.2
- run upstream_tags check on whole bam file, not just forward read, otherwise check fails if there is an inline index in read 2
release 55.1
- bug fix in using roles for finding runfolder path
release 55.00
- schema and bindings re-generated from an copy of a live database
with db updates applied
- verify_bam_id autoqc check added
- changes to database schema, code, tests and test data to enable a database
switch to sql strict mode, in particular,
bam_flagstats check to treat library size -1 as undefined,
bustard summary loader to use zeros where N/A value is in the summary file
- DBIx Result class for the fastqcheck table and a loader for fastqcheck files changed
to accommodate a switch to the InflateColumn plugin, which is used for
other classes
- DBIx schema loader:
removed dependency on the soon to be discontinued npg_common::config module;
custom post-generation filter to make generated classes comply with perlcritic
binding generated as Moose classes
- in/de-flators are set for all relevant columns for autoqc results tables
- swich from Fey to DBIx ORM in a database loader for autoqc data
- call_gtck_composite_rpt.pl script moved from lib/* to bin directory; its
dependency on srpipe::runfolder removed
- standard way to run perl critic tests - on all perl modules and scripts in bin
release 54.11
- changes to allow use of multiple SNP sets with the genotype check
- changes to role for pulldown_metrics check to format percentage values correctly and consistently
release 54.10
- use mocked jar file
- podcoverage test runs for all files
- during build, change shebang line of cgi script to perl interpreter
used similar to Build.PL default for scripts
- a reg expr fix in the tags_reporters autoqc check
release 54.9
- add tags_reporters to IGNORE_AUTO_QC_CHECK list in DB.pm to allow runs to be deletable without results for that check.
release 54.8
- add new tags_reporters check
release 54.7
- upstream_tags check: correct determination of archive_qc_path, use position when fetching qc_store results, use db_lookup attribute with qc_store
release 54.6
- randomise order in which references are used by ref match (to work around Lustre client caching bug)
- upstream_tags check uses qc_store to fetch tag information for upstream runs instead of st::api::lims
release 54.5
- corrections to upstream_tags check 1) fix can_run condition, and 2) use CLASSPATH correctly (remove hard-coded path)
release 54.4
- correct upstream_tags check to use the tag indexes from the decode tag set when counting perfect_matches in previous runs
release 54.3
- correct path determination from path attribute to work correctly with value supplied by pipeline
release 54.2
- changes to upstream_tags check
added 5 base tag set (probably should be removed when TraDIS detection is added)
added a java_xmx_flag attribute for BamIndexDecoder invocation (default: -Xmx1000m)
use path attribute (set from qc_in in the pipeline) to locate both tag#0 bam file and for location of BamIndexDecoder metrics file output
added min_percent_match attribute to allow this cutoff to be set at runtime if needed
added recal_path attribute, use it to construct paths to tag0_bam_file and metrics_output_file
changed can_run to detect !lims->is_pool (only run when lane is plexed)
release 54.1
- changes to upstream_tags check
unset NPG_WEBSERVICE_CACHE_DIR to avoid relying on cached data for lims lookup of tag set information
changed parameters of BamIndexDecoder run to allow one mismatch (and added attributes to allow this and the MAX_MISMATCHES parameter to be reset by the caller as required)
release 54.0
- cache_list_query_movez_tiles removed
reason - last run loded to move_z table is 5302
- npg_qc::api::loader::Qcal_Report removed;
reason - table qcal last run loaded 6918
- npg_qc::api::loader::Run_Config removed;
reason - table run_config last run loaded 6927
- npg_qc::api::loader::Tile_Score removed;
lots of tables error_.., errors_.., most_.. last run loaded 6927
- new namesopace for illumina-data related modules - npg_qc::illumina
- npg_qc::api namespace removed, modules that are in use moved to npg_qc::illumina
- where DBI connection is used, both AutoCommit and mysql_auto_reconnect are switched on
- bin/npg_qc_api_monitoring.pl removed, its functionality merged into npg_qc_illumina_analysis_loader
- runfolder checking removed for run loader since it's called from the pipeline,
for loading all runs the globs of runfolders come from the tracking database
- to avoid post requests from 'run_is_deletable' script, npg_qc::illumina::loader module
implements npg-qc-related functionality needed for this script (lane_summary_saved method)
release 53.5
- add functional upstream_tags check
release 53.4
- patch release to add upstream_tags check to IGNORE_AUTO_QC_CHECK list of DB.pm to run to be deletable without this checks results.
release 53.3
- added new upstream_tags check. Check itself currently non-functional, but results should be displayed in Seq-QC pages
release 53.2
- changes to remove most regular warnings when rendering the pages
release 53.1
- changes needed to move NPG QC web server from intweb to webteam's VMs
- lightbox for IVC plots dropped
release 53.0
- notification of perlcritic test failure: include policy name
- to eliminate a separate db configuration file for the code running
on the farm, autoqc & api db interface gets db credentials from a dbix connection
- autoqc & api db-related tests deploy and populate db through dbix binding, take
db credentials from ~/.npg directory
- unused code removed from test util module, test api util module removed
release 52.7
- pulldown_metrics: error message fix and criterion description addition
release 52.6
- dbix binding updated
release 52.5
- ref match check - cache abs path of the default strain/version to avoid being cought out when
the link is switched to a different strain/version
- pulldown metrics check: check now fails when result->on_bait_bases_percent < 20 and when bait_path is found but interval files are not found. Otherwise pass remains undefined.
release 52.4
- removed from Build.PL dependency that installs as a part of other package
- ensure 'use Readonly'; and 'Readonly::Scalar our $VERSION' are on one line to help
with inferring the version
- amend pulldown_metrics check
- "can_run" allows missing baits intervals files
- "execute" fails the check when the baits intervals files are missing
- if the intervals files (bait and target regions) are identical, this is flagged in the result. Seq-QC will be amended to detect this.
- added "pass" and "interval_files_identical" columns to pulldown_metrics table in npg_qc database
release 52.3
- extended the build file to deploy qc bias R script
- made id_run and position attributes optional in genotype check
- added flag to allow specification of reference and position to SNP name mapping file
release 52.2
- genotype check alternative match bug fix
- gc bias check command string bug fix
release 52.1
- pulldown metrics check bug fix - use correct function name for getting picard jar version
release 52.0
- explicit dependency on /software amd /software/solexa removed
- autoqc checks use software_location role to access third party tools
- following loading failure under perl 5.14.2 on precise,
illumina analysis loader code uses standard DBIx transaction
- illumina analysis loader and monitor use standard way of getting DBIx connection
- unused npg_qc::api::run namespace removed
- unused scripts and one-off code removed
- some tests improved, made more robust
- package is build with Build.PL
- all scripts that have to be deployed are moved from scripts to bin
release 51.9
- mysql client does not read configuration in default places to avoid readign wrong configuration
- use password for the root user on a local test database
- removed old unused scripts and modules
- moved window_depth.d source file from bin to src
- removed the specification of the samtools_path from call_gtck_composite_rpt.pl (default should be OK), and changed the name of the requested samtools executable to "samtools_irods".
release 51.8
- tweak spatial filter to deal with output when reads removed rather than marked with fail bit
release 51.7
- Fey2DBIxRedesign document added
- changes to allow genotype check to be run on more than one bam file
- temporary exclusion of Chlorocebus aethiops reference from ref_match check
release 51.6
- bug fix - cope with spatial_filter classes (without tag_index) cat run deletion check time
release 51.5
- bug fix - custom type name update
release 51.4
- reflect that a number of npg_common mudules has been moved to the npg_tracking namespace
- some compatibility changes for perl 5.14.2 on precise
- insert size check can_run method relies only on npg::api::run when
determining whether the run has paired reads
release 51.3
- add spatial_filter to test to ignore when checking for deletion