|
51 | 51 | "\n", |
52 | 52 | "\n", |
53 | 53 | "\n", |
54 | | - "4. If you want to make changes to the code and have your own copy of it to register, clone the code into a public repository in Github or in MAAP Gitlab.\n" |
| 54 | + "While developing an algorithm, this is how you would manage pushes and pulls to and from GitHub. For the purpose of this guide, there is no need to push or pull changes.\n" |
55 | 55 | ] |
56 | 56 | }, |
57 | 57 | { |
|
63 | 63 | "\n", |
64 | 64 | "After creating your MAAP account, you can create a code repository by navigating to the MAAP GitLab account at https://repo.maap-project.org. This GitLab account is connected to your ADE workspaces automatically when signing into the ADE.\n", |
65 | 65 | "\n", |
66 | | - "You can then follow the same steps above to clone a repository from the MAAP GitLab. \n" |
| 66 | + "You can then follow the same steps above to clone a repository from the MAAP GitLab. \n", |
| 67 | + "\n", |
| 68 | + "Typically, scientists have been storing code in GitHub, but the built-in GitLab is available if you prefer.\n" |
| 69 | + ] |
| 70 | + }, |
| 71 | + { |
| 72 | + "cell_type": "markdown", |
| 73 | + "metadata": {}, |
| 74 | + "source": [ |
| 75 | + "## Testing an Algorithm the Workspace\n", |
| 76 | + "\n", |
| 77 | + "To make sure that an Algorithm is functioning as expected, we can run it in the Jupyter terminal in a way to mimic how the scaled DPS (Data Processing System) will run it.\n", |
| 78 | + "\n", |
| 79 | + "To do this, you will want to run the `run-test.sh` script from another folder. For the sake of this test, let's try running it in `/tmp/my_test_run`. You can call the folder whatever you'd like, maybe with your name in it to make it unique (e.g. `/tmp/robs_test_run`).\n", |
| 80 | + "\n", |
| 81 | + "When the DPS runs an algorithm, it will first copy input files into a folder called `input/`. It will write outputs into a folder called `output/`. In order to mimic this, let's copy a test file from the `dps-unit-test` repo into a temporary test-run folder `/tmp/my_test_run/input`. (When we get to the [actual run in the DPS](running_at_scale.ipynb#Run-the-Algorithm-as-a-Job-and-Monitor-it), you will specify an input file URL and that file will be downloaded into the `input/` folder of the cloud-worker before execution of the algorithm.)\n", |
| 82 | + "\n", |
| 83 | + "If you cloned the demo repository to `~/algorithms`, then the files should be somewhere like `~/algorithms/dps-unit-test`. For the sake of the example below, we will assume this is where the demo repo has been cloned.\n", |
| 84 | + "\n", |
| 85 | + "First copy the example input file into a folder called `input/`, from inside your temporary test-run folder.\n", |
| 86 | + "```\n", |
| 87 | + "mkdir /tmp/my_test_run\n", |
| 88 | + "cd /tmp/my_test_run\n", |
| 89 | + "mkdir input\n", |
| 90 | + "cp ~/algorithms/dps-unit-test/input_file.txt input\n", |
| 91 | + "ls input\n", |
| 92 | + "```\n", |
| 93 | + "\n", |
| 94 | + "This should show you the contents of the `input/` folder is the file `input_file.txt`.\n", |
| 95 | + "\n", |
| 96 | + "Then we can execute a test run of the `run-test.sh` script found in the repository you cloned.\n", |
| 97 | + "```\n", |
| 98 | + "cd /tmp/my_test_run\n", |
| 99 | + "~/algorithms/dps-unit-test/run-test.sh\n", |
| 100 | + "```\n", |
| 101 | + "\n", |
| 102 | + "If the run was successful, you should see the following output:\n", |
| 103 | + "```\n", |
| 104 | + "Testing writing output product\n", |
| 105 | + "Testing opening input file\n", |
| 106 | + "Opening input file input/input_file.txt success\n", |
| 107 | + "```\n", |
| 108 | + "\n", |
| 109 | + "Then if you do `ls *` you should see the content of the `input/` and `output/` folders:\n", |
| 110 | + "```\n", |
| 111 | + "> ls *\n", |
| 112 | + "input:\n", |
| 113 | + "input_file.txt\n", |
| 114 | + "\n", |
| 115 | + "output:\n", |
| 116 | + "write-output.txt\n", |
| 117 | + "```\n", |
| 118 | + "\n", |
| 119 | + "If the run was not successful, you may not have created the input folder with the input file in it. This may also happen if you run the `run-test.sh` script from the a different folder (where there is no `input/` folder). In that case you will see an error something like this:\n", |
| 120 | + "```\n", |
| 121 | + "ls: cannot access 'input/*': No such file or directory\n", |
| 122 | + "Testing writing output product\n", |
| 123 | + "Testing opening input file\n", |
| 124 | + "Traceback (most recent call last):\n", |
| 125 | + " File \"/projects/algorithms/dps_unit_test/dps-unit-test/test-input-file.py\", line 6, in <module>\n", |
| 126 | + " input_file = sys.argv[1]\n", |
| 127 | + " ~~~~~~~~^^^\n", |
| 128 | + "IndexError: list index out of range\n", |
| 129 | + "```\n" |
67 | 130 | ] |
68 | 131 | }, |
69 | 132 | { |
70 | 133 | "attachments": {}, |
71 | 134 | "cell_type": "markdown", |
72 | 135 | "metadata": {}, |
73 | 136 | "source": [ |
74 | | - "## Customizing your workspace environment\n", |
| 137 | + "## Customizing your Workspace Environment\n", |
75 | 138 | "\n", |
76 | 139 | "Your Jupyter workspace has a set of pre-installed libraries, depending on [which Stack you selected](getting_started.ipynb#Creating-a-workspace). If you need libraries that are not pre-installed, we suggest using an environment manager; `conda` is pre-installed to help with this. \n", |
77 | 140 | "\n", |
|
86 | 149 | "\n", |
87 | 150 | "The MAAP platform offers a variety of functionality to run and monitor large-scale processing jobs. Access to the functionality is gained via the underlying [RESTful MAAP API](https://api.maap-project.org/api/). In a Python notebook, you will typically use this API via a helper library called `maap.py`, which will make using MAAP platform features easy, using Python syntax. For example, registering algorithms, running batches of jobs, monitoring jobs, or accessing data.\n", |
88 | 151 | "\n", |
89 | | - "Much of the `maap.py` functionality is documented in the [Technical Tutorials section](../technical_tutorials.rst) and in-context in the [Science Tutorials](../science_examples.rst). The [maap-py Github page](https://github.com/MAAP-Project/maap-py) has additional usage documentation.\n" |
| 152 | + "Much of the `maap.py` functionality is documented in the [Technical Tutorials section](../technical_tutorials.rst) and in-context in the [Science Examples](../science_examples.rst). The [maap-py Github page](https://github.com/MAAP-Project/maap-py) has additional usage documentation.\n" |
90 | 153 | ] |
91 | 154 | }, |
92 | 155 | { |
|
105 | 168 | " " |
106 | 169 | ] |
107 | 170 | }, |
| 171 | + { |
| 172 | + "cell_type": "markdown", |
| 173 | + "metadata": {}, |
| 174 | + "source": [ |
| 175 | + "## Registering the Algorithm\n", |
| 176 | + "\n", |
| 177 | + "This section briefly demonstrated how you will sync algorithm code with GitHub as you develop it. Once your algorithm is ready to be run at scale, you will need to Register it with the MAAP DPS.\n", |
| 178 | + "\n", |
| 179 | + "The next section covers this process." |
| 180 | + ] |
| 181 | + }, |
108 | 182 | { |
109 | 183 | "cell_type": "markdown", |
110 | 184 | "metadata": {}, |
|
0 commit comments