Official announcements

MVP phase coming to a close!

Hi all, We're excited by the frenzy of progress being made on the project over the last few weeks. Some logistical notes for how this phase will get closed out: * PRs that are opened up until the MVP phase officially closes at midnight UTC tonight will be eligible for points on the MVP leaderboard. Naturally, that means the MVP leaderboard will be subject to change for a few days after the phase ends as the PRs are resolved. * We will treat these PRs normally and they will go through the customary review process — as usual, our policy within the open source side of the house is to pretend the points don't exist! — except that we expect contributors to be extremely responsive in addressing questions or requested changes so that we can finalize the MVP leaderboard in a timely fashion. * Once the phase ends, the milestone tag on Github will be switched over for issues that now properly belong in the feature building phase. This may not happen instantaneously, but no new points will be entered for the MVP phase after midnight UTC regardless of the Github milestone. For a reminder of how prizes will be allocated at the end of a milestone, see the ['Prizes' page](https://concepttoclinic.drivendata.org/prizes). Happy development! - The C2C team
3 weeks, 2 days ago

Private Docker Cloud repos for early contributors

Just to sweeten the deal: the first 100 submitters of accepted PRs will earn the submitter 5 private Docker Cloud repos thanks to the generous sponsorship of our friends at Docker! Check out the Docker Cloud [site](https://cloud.docker.com/) to learn more about what you can do with private repositories.
3 months ago

Launch Announcement

The Concept to Clinic challenge is live! Thanks for joining in this community effort to bring advances in AI to the front lines of lung cancer detection. Read up on the challenge and Sign Up to start contributing today!
3 months, 2 weeks ago

Newsfeed

@WGierke commented on PR #218: #160 Add Route Testing

@lamby Yes ;) It should be done now.
32 minutes ago

@WGierke commented on issue #229: Algorithms don't work across both dicom and luna scans

Once #237 is merged, I'd like to work on "segmentation test(s) should pass for luna scans".
1 hour, 12 minutes ago

@WGierke opened a new pull request: #237: #187 Fix Segmentation + #151 Training Data Shape

The problem was that we haven't had a standardized way to handle training data. Now, the scans and binary nodule images have the shape 512x512x1024 so all DICOM images that have been rescaled to voxels fit into that. However, since this is a giant input matrix which can easily blow up the memory when training a neural network I added a `SegmentationModel` wrapper that takes this shape as an input and also predicts this shape, but the models that implement this interface can also rescale / crop / ... the input data as long as the predicted output is again of size 512x512x1024. For example consider the [`Simple3DModel`](https://github.com/concept-to-clinic/concept-to-clinic/compare/master...WGierke:187_fix_segmentation?expand=1#diff-7ebde87c389dab64f0c9e8a81e6b6391R49): it scales the input by 1/4 for each axis, learns a model based on that and after predicting it scales the predicted binary nodule mask again by factor 4 such that the output shape equals the given training data shape again. Note that I mainly moved the code from `models.py` to the classes in the `models/` directory to accomplish this. ## Reference to official issue This references #187 and #151 since I wasn't able to clearly separate them. ## How Has This Been Tested? I added the piece of code that threw an error given in #187 as a test. All tests pass. ## CLA - [X] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
4 hours, 3 minutes ago

@Serhiy-Shekhovtsov commented on PR #233: #145 import image series

Image loading is also a part of [this PR](https://github.com/concept-to-clinic/concept-to-clinic/pull/236) that also shows currently running case, allows user to import and start new case after preview image, as it was described in [documentation](http://concept-to-clinic.readthedocs.io/en/latest/design-doc.html#imagery-interface-frontend).
5 hours, 4 minutes ago

@Serhiy-Shekhovtsov commented on PR #226: Implement 'Detect and Select' component (#148)

@lamby, @reubano is there anything blocking the merge of this PR?
5 hours, 7 minutes ago

@Serhiy-Shekhovtsov commented on PR #233: #145 import image series

@vessemer, I think it's a good idea to let others know when you are working on an issue (like I did [here](https://github.com/concept-to-clinic/concept-to-clinic/issues/145#issuecomment-344731667)) if we are choosing collaboration over competition. This way team members will not waste their time working on same tasks.
5 hours, 28 minutes ago

@Serhiy-Shekhovtsov opened a new pull request: #236: Implemented Imagery selection screen

## Description According to [documentation](http://concept-to-clinic.readthedocs.io/en/latest/design-doc.html#imagery-interface-frontend) on this screen should in particular: - List available DICOM images on local filesystem. - Display a preview of an available DICOM image. - Allow user to select a given image and **start a new “Case.”** This image will be used in the identification step next. All these features has been implemented in this PR(frontend and backend). - Now user can see details about currently running case - Can see list of directories and preview an image - Can start a new case for selected image ![start a new case](http://g.recordit.co/Htja5hXlvc.gif) ## Reference to official issue Related to #145 and #202 ## CLA - [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
5 hours, 49 minutes ago

@Serhiy-Shekhovtsov commented on issue #202: Improve open imagery "Import" button UX

!["Import" button UX](http://g.recordit.co/QvlvS3OCv6.gif)
8 hours, 48 minutes ago

@Serhiy-Shekhovtsov opened a new pull request: #235: fixed Vue.js compilation error by updating Vue.js version

Because of [this bug](https://github.com/vuejs/vue/issues/7084) in version 2.4.2 of **Vue.js** the compilation of Vue app was broken. I have fixed it by updating the package. ## CLA - [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
9 hours, 16 minutes ago

@Serhiy-Shekhovtsov commented on PR #232: fixed Vue.js startup issue

@lamby after checking this again I have found that it's a Vue.js bug and it has been fixed in latest version. Will fix my PR. You can check more details [here](https://github.com/vuejs/vue/issues/7084), but basically it was throwing an error when trying to pass `v-model="..."` and `:value="..."` same time to one element.
9 hours, 17 minutes ago

@louisgv commented on issue #202: Improve open imagery "Import" button UX

@Serhiy-Shekhovtsov For sure! I thought you were looking to avoid conflicting with this issue, but if you can solve both that'd be great :p I didn't mean to raise conflict/confusion. Also I was spending the entire day updating my build... refer to #234 :skull:
12 hours, 20 minutes ago

@louisgv commented on issue #234: Docker Build failed, /proc file permission denied

After spending an entire day trying all kind of thing to fix the problem, from reinstalling docker, to remove docker tunnel, to reboot my machine a dozen of time, I decided to just comment out the 8th docker command in the `base` instruction, and that solved the problem. This was super frustrating. But I think this line should be investigated and might need improvement: ``` RUN find . -type d -name __pycache__ -exec rm -r {} \+ ``` Maybe we can use a python package like pyclean or py3clean to clean up the cache instead of searching for it manually?
12 hours, 24 minutes ago

@lamby commented on PR #232: fixed Vue.js startup issue

For someone unfamiliar with Vue.js, can you describe what the issue is here in more detail? :)
13 hours, 44 minutes ago

@louisgv created a new issue: #234: Docker Build failed, /proc file permission denied

<!--- Provide a general summary of the issue in the Title above --> ## Expected Behavior <!--- Tell us what should happen --> Upon pulling new change from master, I should be able to run `docker-compose -f local.yml build` to update the docker image. ## Current Behavior <!--- Tell us what happens instead of the expected behavior --> The build process failed: ``` Step 8/21 : RUN find . -type d -name __pycache__ -exec rm -r {} \+ ---> Running in 41c5736fc3df find: './proc/1/map_files': Operation not permitted find: './proc/7/map_files': Operation not permitted ERROR: Service 'base' failed to build: The command '/bin/sh -c find . -type d -name __pycache__ -exec rm -r {} \+' returned a non-zero code: 1 ``` ## Possible Solution <!--- Not obligatory, but suggest a fix/reason for the bug, --> ## Steps to Reproduce <!--- Provide a link to a live example, or an unambiguous set of steps to --> <!--- reproduce this bug. Include code to reproduce, if relevant --> 1. Pull from master 2. Run `docker-compose -f local.yml build` ## Context (Environment) <!--- How has this issue affected you? What are you trying to accomplish? --> <!--- Providing context helps us come up with a solution that is most useful in the real world --> This might be specific to Debian. My operating system is: ``` Linux lab 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u3 (2017-08-15) x86_64 GNU/Linux ``` ## Detailed Description <!--- Provide a detailed description of the change or addition you are proposing --> ## Possible Implementation <!--- Not obligatory, but suggest an idea for implementing addition or change --> ## Checklist before submitting - [+] I have confirmed this using the officially supported Docker Compose setup using the `local.yml` configuration and ensured that I built the containers again and they reflect the most recent version of the project at the `HEAD` commit on the `master` branch - [+] I have searched through the other currently open issues and am confident this is not a duplicate of an existing bug - [ ] I provided a **minimal code snippet** or list of steps that reproduces the bug. - [ ] I provided **screenshots** where appropriate - [ ] I filled out all the relevant sections of this template
15 hours, 34 minutes ago

@vessemer opened a new pull request: #233: #145 import image series

Import image series implementation. ## Reference to official issue This addresses #145 ## Screenshots (if appropriate): ![peek 2017-11-19 02-36](https://user-images.githubusercontent.com/9470024/32986501-6412673e-ccd3-11e7-954e-f0dc303b8f98.gif) ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
16 hours, 12 minutes ago
18 hours, 26 minutes ago

@Serhiy-Shekhovtsov commented on issue #202: Improve open imagery "Import" button UX

@louisgv , but I already started #202. And this one looks like really minor and will be closed as well.
19 hours, 33 minutes ago

@Serhiy-Shekhovtsov commented on issue #202: Improve open imagery "Import" button UX

I am going to do the following: - Show if there is any case started. If started - show a message that starting a new case will drop the current one. - Show list of images in local system. User can click on image name to view it. (this is already implemented) - On "start case" button click - I will drop the current case and create a new one with selected image.
19 hours, 35 minutes ago

@louisgv commented on issue #202: Improve open imagery "Import" button UX

Hmm, seems like #204 and #202 (this) will be sharing some higher order state. I will do some experiment with this UI element as well as implementing a parent state (to keep track of the step for #204) then :-?
19 hours, 35 minutes ago

@louisgv commented on issue #202: Improve open imagery "Import" button UX

@Serhiy-Shekhovtsov After reading the requirement again, I think we should separate the two function into 2 UI elements (button), one for importing image and one for expanding the image. So maybe for #145, you can simply change the callback of the import button to do what need to be done, or you can also add a new button to import (to avoid conflict with this issue), and then we can refactor the UI at a later stage?
19 hours, 39 minutes ago

@Serhiy-Shekhovtsov commented on issue #202: Improve open imagery "Import" button UX

I am not sure if we should allow selecting multiple images. In documentation we can find the following: > Allow user to select a given image and start a new “Case.” This image will be used in the identification step next Also it will complicate some things, including this story: Prevent progress to next stage of analysis until "completed" the current step (#204). Please, let me know what do you think? (This issue is blocking me with #145)
20 hours, 11 minutes ago

@Serhiy-Shekhovtsov opened a new pull request: #232: fixed Vue.js startup issue

Front-end application was not working because of the following error: `:value conflicts with v-model on the same element because the latter already expands to a value binding internally` ## CLA - [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
20 hours, 45 minutes ago

@musale commented on issue #230: Ask a Clinician! (add a question, get points)

I have some here too: 1. What would you like improved/ changed in your current workflow using tools with standard PACS software package with the current workflow in the concept-to-clinic project? 2. In the lung cancer screening report, is there information in the standard template that is missing and you would want in the report generated by concept-to-clinic project (or omitted)? If there is, which info and why? 3. In the event the tool being built would allow multiple algorithm results and reports were based on a given algorithm, how would that impact the final decision a clinician makes about the cancer detection?
23 hours, 49 minutes ago

@musale commented on issue #227: Fix interface building

@reubano interestingly, I've done the build without any errors from the current master branch. **OS** : Manjaro Linux **Docker version**: 17.10.0-ce, build f4ffd2511c My suggestion would be you try with upgraded docker?
1 day, 1 hour ago

@isms commented on PR #231: Fixed interface API to return relative urls instead of absoulte

@Serhiy-Shekhovtsov Couple minor changes requested but other than that looks great. Thanks!
1 day, 3 hours ago

@Serhiy-Shekhovtsov commented on issue #228: Any reason for having HyperlinkedModelSerializer instead of ModelSerializer?

@isms, your suggestion to inherit all from base and override the `get_serializer_context` is nice, didn't think about it. And after I have implemented it the issue with `.json` is also gone. Btw, here are my findings about `.json` issue: When request is *not* set to None for the serializer, then generated URL will be absolute **and also** copy the extension of the query. If you load `/api/candidates.json` each candidate will have URL `xxx/api/candidates/1.json`.
1 day, 3 hours ago

@Serhiy-Shekhovtsov opened a new pull request: #231: Fixed interface API to return relative urls instead of absoulte

Now each `[Model]ViewSet` will inherit from `ViewSetBase` and that base view class has `get_serializer_context` method overridden to set `'request': None` for the serialization context. ## Description After this change serialized entities have relative URLs instead of absolute and it makes them usable from the Vue.js application. Problem with absolute URLs - interface API is hosted on different port number then *Vue.js* and requests to that port number being blocked by browser because of CORS policy. Now the URLs are pointing to the same host and will be redirected by *Node.js* server. Fix suggested @isms when discussing #228 ## CLA - [ ] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
1 day, 5 hours ago

@WGierke commented on issue #230: Ask a Clinician! (add a question, get points)

I have some pretty obvious questions: 1. If you could wish for a piece of software that could help you detecting lung cancer nodules, what would be its most useful features and how would it be different from any possible "competitors" that already exist? 2. What would be the most important properties such a software has to fulfill such that you would use it on a regular basis (intuitive user interface, responsive, high accuracy, ...?)
1 day, 9 hours ago

@louisgv commented on issue #230: Ask a Clinician! (add a question, get points)

1. How often does radiologists spot error in patient's files/data and what kind of tool do they currently using to validate data and/or fix error? 2. In the Report/Export view, would clinicians prefer to have as much data cramped into the view-port as possible (for quick scanning, for example) in a way similar to infographic report (example)[http://myonlinecareerspace.com/blog/wp-content/uploads/2010/02/educating-future-workforce-education-infographic.jpg] or would they prefer spacial line-by-line document format (example)[https://images.template.net/wp-content/uploads/2015/05/Medical-Report-Template-Free-Download.jpg]? (Specifically talking about the RSNA standard) 3. > NOTE: The 2nd question targets specifically for digital viewing. Exported file will be formatted to follow the formal convention.
1 day, 12 hours ago
1 day, 13 hours ago
1 day, 17 hours ago

@lamby commented on PR #226: Implement 'Detect and Select' component (#148)

LGTM, but would love some further eyes on this frontend stuff! @reubano ? :)
1 day, 18 hours ago

@lamby commented on PR #218: #160 Add Route Testing

Getcha. So to summarise, I understand the need to timeout in the case that the port never materialises, but it would be still be great to *start* the tests before this 60 seconds if the ports are open. ie. would you be okay pushing ahead with https://github.com/concept-to-clinic/concept-to-clinic/pull/218#issuecomment-343729340 ? :)
1 day, 18 hours ago

@isms commented on issue #228: Any reason for having HyperlinkedModelSerializer instead of ModelSerializer?

Hyperlinked fields let the frontend get canonical URLs from the backend rather than doing `domain + id + endpoint` stuff in frontend JS which will immediately break if the endpoints change or if a field other than `id` is used for lookups in the future. Instead the idea is that you can fetch a related resource by fetching `object.url`. You can make it work either way but including links seems to be considered truer to the REST concept. Two address the two specifics: 1. "Also the url contains .json extension" - is there an example of an issue? Or is this a setting that can simply be changed on the serializer or view? Seems like views are still free to return custom endpoints like `/mark` and `/dismiss` that don't go by this scheme as the only thing HyperlinkedModelSerializer tries to add links for is the specific instance it is serializing. 2. Looks like we may be able to get relative URLs simply by having the `[Thing]ViewSet` views in `backend/api/views.py` all extend a class which extends the `viewsets.ModelViewSet`'s [get_serializer_context](http://www.cdrf.co/3.1/rest_framework.viewsets/ModelViewSet.html#get_serializer_context) and overriding the parent context making sure `request=None` (ref: https://github.com/encode/django-rest-framework/issues/3532#issuecomment-223301559) Is there are strong reason to prefer reverting to non-hyperlinked versions?
1 day, 22 hours ago

@pjbull created a new issue: #230: Ask a Clinician! (add a question, get some points)

This project is about making something useful for clinicians. This is your chance to pull up and get input from radiologists working on the front lines of early detection. We would love to hear what you are wondering about in terms of the workflow at the clinic, and where user input can address thoughts that are coming up for you as you work on the application. - What are your questions about a radiologist's process? - What are your questions about what information is most valuable to see? - What are your questions about how the algorithms will be used? - What are your questions about how the interface should work? The team at ALCF will be coordinating responses to submitted questions through their network of clinicians, researchers, and patients. Submit a question by commenting directly on this issue. **You'll earn 2 community points for each of up to three original, substantive questions.**
1 day, 22 hours ago

@reubano created a new issue: #229: Algorithms don't work across both dicom and luna scans

## Overview Currently, the classification algorithm is tested with a luna scan, and the segmentation and identification algorithms are tested with dicom scans. This indicates that there may be some incompatibilities between dicom and luna scans that make them not work with certain algorithms. I have confirmed this by adding new tests for the complementary scans and produced the following results: algorithm|command|dicom|luna ----------|---------|-------|---- classification|`docker-compose -f local.yml run prediction pytest -vrsk src/tests/test_classification.py`|FAILED|PASSED segmentation|`docker-compose -f local.yml run prediction pytest -vrsk src/tests/test_segmentation.py`|PASSED|FAILED identification|`docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_dicom_001`|*crashes*|N/A identification|`docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_dicom_003`|*crashes*|N/A identification|`docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_luna`|N/A|FAILED ## Expected Behavior All algorithm tests should pass ## Technical details Below are the specific tracebacks: `test_classification.py` ```bash (alcf) ⚡ docker-compose -f local.yml run prediction pytest -vrsk src/tests/test_classification.py ========================================== test session starts ========================================== platform darwin -- Python 3.6.2, pytest-3.1.3, py-1.4.34, pluggy-0.4.0 -- /opt/local/bin/python cachedir: .cache rootdir: /Users/reubano/Documents/Projects/alcf/prediction, inifile: collected 3 items src/tests/test_classification.py::test_classify_predict_load PASSED src/tests/test_classification.py::test_classify_dicom FAILED src/tests/test_classification.py::test_classify_luna PASSED =============================================== FAILURES ================================================ __________________________________________ test_classify_dicom __________________________________________ dicom_paths = ['/Users/reubano/Documents/Projects/alcf/images_full/LIDC-IDRI-0002/1.3.6.1.4.1.14519.5.2.1.6279.6001.4901573811602007...14519.5.2.1.6279.6001.298806137288633453246975630178/1.3.6.1.4.1.14519.5.2.1.6279.6001.179049373636438705059720603192'] nodule_locations = [{'x': 50, 'y': 50, 'z': 21}] model_path = '/Users/reubano/Documents/Projects/alcf/prediction/src/algorithms/classify/assets/gtr123_model.ckpt' def test_classify_dicom(dicom_paths, nodule_locations, model_path): > predicted = trained_model.predict(dicom_paths, nodule_locations, model_path) src/tests/test_classification.py:9: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ src/algorithms/classify/trained_model.py:40: in predict return gtr123_model.predict(dicom_path, centroids, model_path) src/algorithms/classify/src/gtr123_model.py:264: in predict ct_array, meta = preprocess(*load_ct(ct_path)) src/preprocess/load_ct.py:96: in load_ct dicom_pattern = os.path.join(path, '*.dcm') _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ a = ['/Users/reubano/Documents/Projects/alcf/images_full/LIDC-IDRI-0002/1.3.6.1.4.1.14519.5.2.1.6279.6001.4901573811602007...14519.5.2.1.6279.6001.298806137288633453246975630178/1.3.6.1.4.1.14519.5.2.1.6279.6001.179049373636438705059720603192'] p = ('*.dcm',) def join(a, *p): """Join two or more pathname components, inserting '/' as needed. If any component is an absolute path, all previous path components will be discarded. An empty last part will result in a path that ends with a separator.""" > a = os.fspath(a) E TypeError: expected str, bytes or os.PathLike object, not list ../../../../.virtualenvs/alcf/lib/python3.6/posixpath.py:78: TypeError ================================== 1 failed, 2 passed in 12.92 seconds ================================== ``` `test_segmentation.py` ```bash (alcf) ⚡ docker-compose -f local.yml run prediction pytest -vrsk src/tests/test_segmentation.py ========================================== test session starts ========================================== platform darwin -- Python 3.6.2, pytest-3.1.3, py-1.4.34, pluggy-0.4.0 -- /opt/local/bin/python cachedir: .cache rootdir: /Users/reubano/Documents/Projects/alcf/prediction, inifile: collected 5 items src/tests/test_segmentation.py::test_correct_paths PASSED src/tests/test_segmentation.py::test_segment_predict_load PASSED src/tests/test_segmentation.py::test_segment_dicom PASSED src/tests/test_segmentation.py::test_segment_luna FAILED src/tests/test_segmentation.py::test_lung_segmentation SKIPPED ======================================== short test summary info ======================================== SKIP [1] src/tests/test_segmentation.py:38: Takes very long =============================================== FAILURES ================================================ ___________________________________________ test_segment_luna ___________________________________________ metaimage_path = '/Users/reubano/Documents/Projects/alcf/images/LUNA-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797.mhd' luna_nodule = {'x': 0, 'y': 100, 'z': 556} def test_segment_luna(metaimage_path, luna_nodule): > predicted = predict(metaimage_path, [luna_nodule]) src/tests/test_segmentation.py:33: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ src/algorithms/segment/trained_model.py:58: in predict volumes = calculate_volume(segment_path, centroids) src/algorithms/segment/trained_model.py:85: in calculate_volume labels = [mask[centroid['x'], centroid['y'], centroid['z']] for centroid in centroids] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .0 = <list_iterator object at 0x13b1a4f28> > labels = [mask[centroid['x'], centroid['y'], centroid['z']] for centroid in centroids] E IndexError: index 556 is out of bounds for axis 2 with size 128 src/algorithms/segment/trained_model.py:85: IndexError ============================ 1 failed, 3 passed, 1 skipped in 43.85 seconds ============================= ``` `test_identification.py` ```bash (alcf) ⚡ docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_dicom_001 ========================================== test session starts ========================================== platform linux -- Python 3.6.3, pytest-3.1.3, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python3.6 cachedir: .cache rootdir: /app, inifile: collected 1 item src/tests/test_identification.py::test_identify_dicom_001 (alcf) ⚡ docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_dicom_003 ========================================== test session starts ========================================== platform linux -- Python 3.6.3, pytest-3.1.3, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python3.6 cachedir: .cache rootdir: /app, inifile: collected 1 item src/tests/test_identification.py::test_identify_dicom_003 (alcf) ⚡ docker-compose -f local.yml run -e RUN_SLOW_TESTS=true prediction pytest -vrsk src/tests/test_identification.py::test_identify_luna ========================================== test session starts =========================================== platform linux -- Python 3.6.3, pytest-3.1.3, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python3.6 cachedir: .cache rootdir: /app, inifile: collected 1 item src/tests/test_identification.py::test_identify_luna FAILED ================================================ FAILURES ================================================ ___________________________________________ test_identify_luna ___________________________________________ metaimage_path = '/images/LUNA-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797.mhd' luna_nodule = {'x': 0, 'y': 100, 'z': 556} @skip_if_slow def test_identify_luna(metaimage_path, luna_nodule): > predicted = trained_model.predict(metaimage_path) src/tests/test_identification.py:25: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dicom_path = '/images/LUNA-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797.mhd' def predict(dicom_path): """ Predicts centroids of nodules in a DICOM image. Given an iterator of DICOM objects, this method will: (1) load the identification model from its serialized state (2) pre-process the dicom into whatever format the identification model expects (3) return centroids with a probability that each centroid is a nodule (as opposed to not a nodule) Note: This model doesn't detect whether or not a nodule is cancerous, that is done in the ``classify`` model. Args: dicom_path (str): a path to a DICOM image Returns: list(dict): a list of centroids in the form:: {'x': int, 'y': int, 'z': int, 'p_nodule': float} """ reader = sitk.ImageSeriesReader() filenames = reader.GetGDCMSeriesFileNames(dicom_path) if not filenames: message = "The path {} doesn't contain any .mhd or .dcm files" > raise ValueError(message.format(dicom_path)) E ValueError: The path /images/LUNA-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797.mhd doesn't contain any .mhd or .dcm files src/algorithms/identify/trained_model.py:49: ValueError ------------------------------------------ Captured stderr call ------------------------------------------ WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/GDCM/src/itkGDCMSeriesFileNames.cxx, line 75 GDCMSeriesFileNames (0x2ebf770): /images/LUNA-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797/1.3.6.1.4.1.14519.5.2.1.6279.6001.102133688497886810253331438797.mhd is not a directory WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/GDCM/src/itkGDCMSeriesFileNames.cxx, line 122 GDCMSeriesFileNames (0x2ebf770): No Series can be found, make sure your restrictions are not too strong ======================================= 1 failed in 27.62 seconds ======================================== ``` ## Steps to Reproduce In order to replicate these results you must first apply the following [patch file](https://github.com/concept-to-clinic/concept-to-clinic/files/1483329/patch.txt). ```bash git checkout 8380847 git apply patch.txt ``` ## Acceptance criteria - [ ] classification test should pass for dicom scans - [ ] segmentation test should pass for luna scans - [ ] identification test should pass for dicom scans - [ ] identification test should pass for luna scans *NOTE: All PRs must follow the standard PR checklist.*
1 day, 23 hours ago

@reubano commented on PR #221: Update PR TEMPLATE to include prediction algorithm metrics

One metric I left out is an estimation of [model over/under fitting](http://wiki.fast.ai/index.php/Lesson_3_Notes#On_Deeper_Finetuning) ([video](https://www.youtube.com/watch?v=6kwQEBMandw)). This is something we should probably address at some point.
2 days, 1 hour ago

@Serhiy-Shekhovtsov created a new issue: #228: Any reason for having HyperlinkedModelSerializer instead of ModelSerializer?

Is there any reason for using `HyperlinkedModelSerializer` for serializing the data before sending it to client side? ### Having this serializer have following drawbacks: - Returned data doesn't have the **id**. Here is an example of data for `/api/candidates.json` method: `{"url":"http://interface:8001/api/candidates/1.json","centroid":{ ... },"created":" ... ","probability_concerning":0.3,"case":"http://interface:8001/api/cases/1.json"}` - Returned URL is not usable. - As you can see, it's absolute and since the port doesn't match the Vue.js server's port - AJAX requests to that URL are blocked by the browser. - Also the url contains **.json** extension. For example, services we have for marking/dismissing candidates look like this: `candidates/1/mark`, `candidates/1/dismiss` ### So, what are the benefits? Maybe I am missing something here, since I am totally new to django, but it looks like replacing `HyperlinkedModelSerializer` with `ModelSerializer` helps to solve the issue of missing **id** and it also doesn't return unusable urls. ## Possible Solution AFAIK, we can still use `HyperlinkedModelSerializer` but patch it by overriding a method for creating serializer and passing `None` as `request`. Or we can simply use to `ModelSerializer`.
2 days, 21 hours ago
3 days, 3 hours ago

@musale commented on PR #224: #206 IsADirectoryError fix

@lamby I think the update should be clearer on the changed files now 😄
3 days, 4 hours ago

@musale commented on PR #224: #206 IsADirectoryError fix

hi @lamby yes. I meant to squash it into one commit.
3 days, 5 hours ago

@swederik commented on PR #197: #148 Manipulations with DICOM vue

Nice work! Awesome to see Cornerstone added to this project! Sorry about the issues with the README. For what it's worth, we are working towards dropping jQuery (though it will be slow to remove it from Cornerstone Tools since we are using it all over the place for events).
3 days, 7 hours ago

@WGierke commented on issue #195: Stop slow tests after a set timeout period, instead of skipping them

`The identification tests should fail on travis even when RUN_SLOW_TESTS isn't set to true.` Assuming that an error occurs in the seconds before the timeout, right? Otherwise the test passes because the timeout is not seen as an error?
3 days, 10 hours ago

@WGierke commented on PR #218: #160 Add Route Testing

` this is about waiting for things to spin-up, no` Exactly, that's what I meant. Sorry for the confusion :)
3 days, 10 hours ago

@reubano created a new issue: #227: Fix interface building

## Overview Building the interface service fails with the following error ```bash ⚡ docker-compose -f local.yml build interface ... Step 3/9 : RUN sed -i 's/\r//' /entrypoint.sh ---> Running in 4e5814af28da container_linux.go:265: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory" ERROR: compose.cli.main.main: Service 'interface' failed to build: oci runtime error: container_linux.go:265: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory" ``` ## Expected Behavior Interface services should build without error ## Technical details - This bug first appears in #199 - Possibly related to https://github.com/moby/moby/issues/5135 - Possibly specific to Docker For Mac - Docker For Mac v17.09.0-ce-mac35 - macOS 10.13 ``` $ docker --version Docker version 17.09.0-ce, build afdb6d4 ``` ## Acceptance criteria - [ ] `docker-compose -f local.yml build` should complete without error. - [ ] `docker-compose -f local.yml build interface` should complete without error. *NOTE: All PRs must follow the standard PR checklist.*
3 days, 18 hours ago

@lamby commented on PR #225: #205 Export pdf functionality.

I'd love to see an example output? Can you upload something somewhere? :)
3 days, 18 hours ago
3 days, 18 hours ago

@lamby commented on issue #205: Export PDF functionality

@Serhiy-Shekhovtsov Implementing client-side is fine :)
3 days, 18 hours ago

@lamby commented on PR #224: #206 IsADirectoryError fix

Hi @musale! Did you mean to squash these into one commit? If you still have the separate changes on your local branch it would be great if you could push them instead; I'm seeing a bunch of whitespace changes that is making the rest hard to review in terms of what it fixes! No worries if not, but I thought I would just ping first instead of picking it apart manually :)
3 days, 18 hours ago

@lamby commented on PR #218: #160 Add Route Testing

@WGierke Sorry for the delay in getting back to you. Perhaps I'm misunderstanding something - if the port is open and subsequently closes, we are safe to assumine something is FUBAR and fail the test. However, AIUI this is about waiting for things to spin-up, no? Sorry if I'm misunderstanding something here. :)
3 days, 18 hours ago

@dannyrb commented on issue #215: Eliminate/Restrict Jquery use for cornerstone-tools

It's worth noting that cornerstone is on track to drop its dependency on jQuery. Work is actively being done. It might be worth waiting for the official change, and then update to latest.
3 days, 20 hours ago

@Serhiy-Shekhovtsov commented on issue #215: Eliminate/Restrict Jquery use for cornerstone-tools

@truefalse10 btw, looks like they have a branch for this: https://github.com/chafey/cornerstone/tree/zachasme-es6 More details [here](https://github.com/chafey/cornerstone/issues/180) and [here](https://github.com/chafey/cornerstone/issues/92)
3 days, 21 hours ago

@Serhiy-Shekhovtsov opened a new pull request: #226: Implement 'Detect and Select' component (#148)

Implemented draggable marker for Detect and Select page for issue #148 ## Description [vue-draggable-resizable](https://github.com/mauricius/vue-draggable-resizable) component used to implement draggable nodule marker. The marker is implemented as a separate component (open-image\NoduleMarker.vue) it takes `marker` (object with coordinates) param and navigation\zoom params `zoomRate`, `offsetX`, `offsetY` ## Screenshots (if appropriate): ![Detect and Select](http://g.recordit.co/wtf2QJwtYP.gif) ### What is not done: - loading image from backend(it requires a method for loading currently selected case) - saving mark\dismiss\coordinates on backend If someone familiar with backend side would like to jump in and help to complete this page - I am more then happy to help from client side. ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
3 days, 21 hours ago

@musale opened a new pull request: #225: #205 Export pdf functionality.

Clicking on the report and export section, you have an export functionality that downloads the page as a pdf file. ## Description Create a pdf file of the section of the site in the **Report and Export** section ## Reference to official issue Addresses issue #205 ## Motivation and Context This feature allows downloading of the page information. ## How Has This Been Tested? - Load [http://localhost:8080/](http://localhost:8080/) - Click on the **Report and Export** menu section - Click on the **Export** button - It should download the page information in pdf format ## CLA - [X] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
4 days, 21 hours ago

@truefalse10 commented on issue #215: Eliminate/Restrict Jquery use for cornerstone-tools

is anyone working on this right now? otherwise i would start
5 days, 2 hours ago

@musale opened a new pull request: #224: #206 IsADirectoryError fix

Loading [http://localhost:8080/](http://localhost:8080) throws an `IsADirectoryError: [Errno 21] Is a directory: '/'` ## Description The endpoint is called every time the front end loads. Since the `dicom.readfile()` method is an IO op, I do a `try/except` to catch the `IOError` 21. As it is, this error is only anticipated on page load because there's no default path to a dicom file. I return an 'empty' response back so that the error is handled on the server-side and presents the client side an opportunity to handle it too. ### To handle the empty response from the server. `Cornerstone` throws errors if handed empty fields on the front end. Since the functions trigger calls to the server endpoint on load (I am assuming this is because of the `async/await` computed functions -- please correct my reasoning here if I'm wrong) and has nothing to display, catering for instances of empty values mitigates these unnecessary errors. Check the changes in [OpenDICOM.vue file](https://github.com/musale/concept-to-clinic/blob/fb39d63d387f2706c6e69792c3739bf61f76506f/interface/frontend/src/components/open-image/OpenDICOM.vue). ## Reference to official issue Fixes issue #206 ## Motivation and Context This change removes the error logs for `IsADirectoryError` and also handles instances where a dicom file is not found. ## How Has This Been Tested? - Load [http://localhost:8080/](http://localhost:8080) - There should be no error traceback ## CLA - [X] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
5 days, 7 hours ago

@isms commented on PR #197: #148 Manipulations with DICOM vue

Merged. This is fgreat - thanks! :confetti_ball:
5 days, 23 hours ago

@WGierke commented on PR #218: #160 Add Route Testing

Okay but what if a port won't be opened because the service died for some reason? In that case we test endlessly which might be no problem on Travis since there is a timeout but not on local computers.
6 days, 9 hours ago

@lamby commented on PR #218: #160 Add Route Testing

> are ready to be tested Right, can't we just wait for that to happen by testing them? Not sure why we need a timeout for this.
6 days, 9 hours ago

@WGierke opened a new pull request: #223: #222 Fix docker-compose build error

This assigns a static name to the generated `base` image such that docker still works if your directory is named something else than "concept-to-clinic". This addresses #222 . ## CLA - [X] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well
6 days, 9 hours ago

@WGierke commented on issue #222: Fix docker-compose build error

Setting the image name automatically would be the cleanest solution IMO. [SO](https://stackoverflow.com/questions/32230577/how-do-i-define-the-name-of-image-built-with-docker-compose) says `docker-compose --project-name foo build bar` would build the image foo_bar no matter what the parent directory is named. [This answer](https://stackoverflow.com/questions/32230577/how-do-i-define-the-name-of-image-built-with-docker-compose/33951705#33951705) even shows how to do that using the docker-compose YAML file.
6 days, 20 hours ago

@reubano commented on issue #222: Fix docker-compose build error

So is this something that can be set dynamically? Or should we have a disclaimer somewhere that telling people what to do if they encounter this error?
6 days, 20 hours ago
6 days, 20 hours ago

@reubano commented on issue #222: Fix docker-compose build error

```bash $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alcf_base latest ab2fe956c2de 2 days ago 3.14GB ubuntu rolling 79dbcfa8f169 8 days ago 93.8MB ``` Changing the dockerfile to `FROM alcf_base` did the trick, thanks! Guess it goes by the name of the project folder.
6 days, 20 hours ago

@WGierke commented on issue #222: Fix docker-compose build error

Can you give us the output of `docker images | grep concepttoclinic_base` ? I think the base image might get another name at your computer (since it works on Travis).
6 days, 20 hours ago

@reubano created a new issue: #222: Fix docker-compose build error

## Overview Building any docker service fails with the following ```bash ⚡ docker-compose -f local.yml build ... Building prediction Step 1/2 : FROM concepttoclinic_base ERROR: Service 'prediction' failed to build: pull access denied for concepttoclinic_base, repository does not exist or may require 'docker login' ``` ## Expected Behavior Services should build without error ## Acceptance criteria - [ ] `docker-compose -f local.yml build` should complete without error. - [ ] `docker-compose -f local.yml build prediction` should complete without error. *NOTE: All PRs must follow the standard PR checklist.*
6 days, 20 hours ago

@WGierke commented on PR #218: #160 Add Route Testing

:D Okay so how about, instead of `sleep 60`, we try to connect to the 4 ports and after 60 seconds we time out and exit with 1, otherwise we perform the tests?
1 week ago

@lamby commented on PR #197: #148 Manipulations with DICOM vue

Sure. I'd love to get another pair of eyes on this though.. @isms ? :)
1 week ago

@lamby commented on PR #218: #160 Add Route Testing

I really don't like the sleep - can we not test instead? :p
1 week ago

@Serhiy-Shekhovtsov commented on PR #197: #148 Manipulations with DICOM vue

Guys @lamby, @louisgv, these changes are very important for #148 and #150. In fact, I can't proceed with #148 without it. Can we merge the current request and create a new issue for "slimmer version of jquery"? P.S. great job, @vessemer!
1 week, 1 day ago