@lamby done

<!--- Provide a general summary of your changes in the Title above --> Remove all __pycache__ files when you run `docker-compose -f local.yml build prediction`. ## Description <!--- Describe your changes in detail --> Added a one-line code in Docker file referenced at #161 by @reubano ## Reference to official issue <!--- If fixing a bug, there should be an existing issue describing it with steps to reproduce --> <!--- Please link to the issue here: --> fixes #161 ## Motivation and Context <!--- Why is this change required? What problem does it solve? --> <!--- If adding a new feature or making improvements not already reflected in an official issue, please reference the relevant sections of the design doc --> Old __pycache__ files can interfere with proper running of docker-compose -f local.yml build and may even cause the following error: ``` import file mismatch: imported module 'app.src.tests.test_load_ct' has this __file__ attribute: /Volumes/Nahla/alcf/prediction/src/tests/test_load_ct.py which is not the same as the test file we want to collect: /app/src/tests/test_load_ct.py ``` There should be a convenient script that can be run to remove all __pycache__ files. ## How Has This Been Tested? <!--- Please describe in detail how you tested your changes. --> <!--- Include details of your testing environment, and the tests you ran to --> <!--- see how your change affects other areas of the code, etc. --> Run build several times without any error. ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

@lamby sorry for the late.

API implementation for dicom metadata and preview available at: `/api/images/metadata?dicom_location='IMAGE_PATH'`. ## Description Before selecting particular DICOM image for analysis we need to display its metadata with preview. This API provides those data. ## Reference to official issue #33 ## How Has This Been Tested? Test case to check HTTP 200 OK status for valid dicom image has been added. However, because of the issue #116, test cases are failing for the test images. I am adding screenshot which I verified using a dicom image from original dataset. ## Screenshots (if appropriate):  ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

Here what I am trying to do. ``` dicom_dict = {} repr(ds) # <-------------- KeyError: "Unknown DICOM tag (6576, 7372) - can't look up VR" for dicom_value in ds: if dicom_value.tag == (0x7fe0, 0x0010): # discard pixel data continue else: dicom_dict[dicom_value.tag] = self._convert_value(dicom_value.value) ``` Simple loop: ``` for dicom_value in ds: # <-------------- KeyError: "Unknown DICOM tag (6576, 7372) - can't look up VR" pass ```

@WGierke I think it is my fault. read_file is not raising the error. I am calling `repr(ds)` after which is producing the error. However, I can not load image using [ImageJ](https://apps.ubuntu.com/cat/applications/imagej/)

``` path = '/images/LIDC-IDRI-0002/' \ '1.3.6.1.4.1.14519.5.2.1.6279.6001.490157381160200744295382098329/' \ '1.3.6.1.4.1.14519.5.2.1.6279.6001.619372068417051974713149104919/-80.750000.dcm' ds = dicom.read_file(path, force=True) # At first tried without force=True ```

<!--- Provide a general summary of the issue in the Title above --> DICOM images in both full and test images not working with pydicom package. ## Expected Behavior <!--- Tell us what should happen --> DICOM files from LUNA dataset seem to be working fine with pydicom. ## Current Behavior <!--- Tell us what happens instead of the expected behavior --> Raising following error: `dicom.errors.InvalidDicomError: File is missing 'DICM' marker. Use force=True to force reading` I was unable to open it using other DICOM image viewer as well telling same error. Doing `force=True` does not work either and raise inavlid DICOM tag error. ## Possible Solution <!--- Not obligatory, but suggest a fix/reason for the bug, --> May be images copied into test directory corrupted? ## Steps to Reproduce <!--- Provide a link to a live example, or an unambiguous set of steps to --> <!--- reproduce this bug. Include code to reproduce, if relevant --> 1. I tried to open it using pydicom. Error 2. Tried with [ImageJ](https://apps.ubuntu.com/cat/applications/imagej/). Error 3. I tried original LUNA image with above two method. Both works fine. 4. I found mismatch in size between original LUNA and images in full/small as well. ## Context (Environment) <!--- How has this issue affected you? What are you trying to accomplish? --> <!--- Providing context helps us come up with a solution that is most useful in the real world --> I was trying to load metadata information as of #33 ## Possible Implementation <!--- Not obligatory, but suggest an idea for implementing addition or change --> Copy data from original dataset again may be? ## Checklist before submitting - [x] I have confirmed this using the officially supported Docker Compose setup using the `local.py` configuration and ensured that I built the containers again and they reflect the most recent version of the project at the `HEAD` commit on the `master` branch - [x] I have searched through the other currently open issues and am confident this is not a duplicate of an existing bug - [x] I provided a **minimal code snippet** or list of steps that reproduces the bug. - [x] I provided **screenshots** where appropriate - [x] I filled out all the relevant sections of this template

@lamby I have updated the code to support recursive directory traversal. This will now return the directory tree in following format: ``` {'directories': [ { 'name': directory_name1, 'children': [ file_name1, file_name2, { 'name': 'nested_dir_1', 'children': [ 'file_name_1', 'file_name_2', .... ] }, ... ] }, ... ] } ``` 

@lamby It is for later. I ignored deeper directory traverse as per discussion on #85. I can work on it now if it is desirable functionality though.

Return simple 2 level directory listing from `/images/available` api ## Description `/images/available` will return directory and files in alphanumerically sequential order ## Reference to official issue issue #31 ## Motivation and Context Current API return only emplty list. ## How Has This Been Tested? Unit test added to test http response. We have tested against the test image direcotry available at `/images/LIDC-IDRI-0001/1.3.6.1.4.1.14519.5.2.1.6279.6001.298806137288633453246975630178` which return following direcotry list in alphabetical order ## Screenshots (if appropriate): In Index Page:  API returns:  ## Updates of Acceptance Criteria - [x] Correct hierarchical listing - [x] Sorted alphabetically ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

@lamby done

@lamby let me know if there is anything that should be done for this pull request.

Hi @WGierke . One thing that passed on my mind when I initially started working on the issue was - whether the directory structure will be fixed or it will be dynamic. ``` +-- root |-- Level 1 |-- Level 2 directory 1 |-- case 001 |-- case 002 |-- Level 2 directory 2 |-- case 001 |-- case 002 ``` Do we want to grab all the files and directories under the sub directories as well? For instances, LIDC-IDRI data does not follow this structure. Nor the test images directory. If we move to another data source (or DICOM image server which is planned for production) the structure might be different. But, if that is not the case, next step will be very easy. What do you think about the issue? I think @lamby can help us by guiding on this. :)

Thank you @lamby for the suggestions. I could not get the chance to work on the modifications. I would like to let you know that your feedback is clear to me and I will get back asap.

@lamby Resolved conflicts and modified as per your suggestion. I have mistakenly used master of forked repo for the pull request. I will do it in new branch for future changes. :)

<!--- Provide a general summary of your changes in the Title above --> New API for direcotory listing in homepage as described in #77 ## Description Introducing new API endpoint which will be used to retrieve images from data directory. <!--- Describe your changes in detail --> ## Reference to official issue See issue #77 <!--- If fixing a bug, there should be an existing issue describing it with steps to reproduce --> <!--- Please link to the issue here: --> ## How Has This Been Tested? This modification will effect homepage initial dataload. Implementation of the API is yet to be made. <!--- Please describe in detail how you tested your changes. --> <!--- Include details of your testing environment, and the tests you ran to --> <!--- see how your change affects other areas of the code, etc. --> ## Updates of Acceptance Criteria - [x] Update routes - [x] Declare view (Not implementation) - [x] Update index.html API call ## CLA - [x] I have signed the CLA; if other committers are in the commit history, they have signed the CLA as well

As per discussion in issue #31 we need a separate endpoint to fetch dataset containing images and directories. <!--- Provide a general summary of the issue in the Title above --> ## Expected Behavior API call should return a list of directories and images of the dataset without preloading the dataset into the database. <!--- Tell us what should happen --> ## Current Behavior Currently, homepage is making API call to `/api/images/` which is actually a ModelViewSet connected with a database model. <!--- Tell us what happens instead of the expected behavior --> ## Possible Solution A completely separate endpoint needed to be called on load to fetch directory listing. This endpoint will later look into specific server directory and return the list in alphabetical order as mentioned in #31 . <!--- Not obligatory, but suggest a fix/reason for the bug, --> ## Detailed Description Current API call in index page retrieving image list from `/api/images/` which is using magic method of Django REST frameworks's `ModelViewSet` to list down all ImageSeries object. Since `ImageSeries` is a database model, at first the files needed to be loaded into database to instantiate that way. On the other hand, to load files directly from the filesystem, we should use Django's `storage` class as it is mentioned in #31 . This is why we need to use different API endpoint to retrieve files from storage. ## Possible Implementation This issue will only introduce a new API endpoint. Further implementation will be tracked in #31 <!--- Not obligatory, but suggest an idea for implementing addition or change --> ## Acceptance criteria - [ ] Update routes - [ ] Declare view (Not implementation) - [ ] Update `index.html` API call

@lamby Got it. I will start working on it. But I am not clear what did you mean by - > Can you open other issue for this

@lamby I will be right here waiting for you. :musical_note: :)

Thank you for your response @lamby. According to current set up, it seems the system is assuming that image files would be preloaded in the database. Current vue frontend is expecting `ImageSeries`. API call is also made to an endpoint which expected to return all `ImageSeries` objects. If we want to import image information based on user's selection, I think the better option would be using `ViewSet` instead of `ModelViewSet`. In extended `ViewSet`, we would be able to decouple filesystem API from image series information API.

Hello! Current API call in index page retrieving image list from `/api/images/`which is using magic method of Django REST frameworks's ModelViewSet to list down all `ImageSeries` object. Since `ImageSeries` is a database model, at first the files needed to be loaded into database to instantiate that way. On the other hand, to load files directly from filesystem, we should use Django's `storage` class as you mentioned above. I would like to know which would be the desired system flow? Or we can to use different API endpoint to retrieve files from storage. It would be helpful if anybody can give some clarification!