overleaf/services/web
James Allen 662122bb1c CI and local dev environment improvements
The need for this became very noticeable due to the slowness of filesystem access in docker-in-mac, with a full compile taking over a minute for me in docker. Using make to introduce incremental compile makes this near instantaneous outside of docker (if only a few files have changed), and quick enough inside docker.

With incremental compile via make, it compiles quickly enough that re-compiling and restarting the web service automatically when backend files change is quick enough now. This is how the service is run via docker-compose in https://github.com/sharelatex/sharelatex-dev-environment, so it shouldn't be necessary to manually restart the container each time a coffee file changes.

At the moment Jenkins pull web modules in via the GitSCM plugin, but I believe this is creating a dependency in Jenkins, where any commits to any of the modules causes all of the web branches to rebuild. By doing it via our own scripts we can hopefully avoid this. It also creates a build process which is reproducible locally.

**Note that at the moment in this PR all modules pull from `ja-dockerize-dev` branches, but these should be merged first, and this PR updated to point to the master branches before merging**. This is necessary for other changes to build process/docker-compose workflow.

As well as a Makefile for web, there is now a `Makefile.module`. This is copied into each module directory by the top-level Makefile, and is written in a way to be flexible and support unit tests, acceptance tests, front-end js for the ide and main, and the modules `app/coffee` directory, while allowing modules to have some of these missing (not all modules have e.g. acceptance tests, or front-end JS). This will allows us to refine the build process in future, without needing to update the Makefile in each module repo separately (I found this to be a painful part of this development).

This makes web compatible with the docker-compose workflow at https://github.com/sharelatex/sharelatex-dev-environment, where each service is running in its own docker container, with networking managed by docker.

Previously the Makefile was set up to run unit tests in docker with `make unit_tests`. This now just runs them natively. In the CI, they are run in docker anyway (all steps in Jenkins are), and locally, they run fine natively with `npm run test:unit`, or can be run in docker via https://github.com/sharelatex/sharelatex-dev-environment with `bin/run web_sl npm run test:unit`.

Previously we did a lot of juggling with only mounting source files (coffee, less, etc) into the docker container for acceptance tests. This was to avoid creating root owned files if the whole directory was mounted. Now instead the whole web directory is mounted read-only, with the compilation step done outside of the container before running the tests.

This allows the host and container to share the `node_modules` folder as well, which avoids needing to `npm install` twice on the CI box, and should speed up the build by a few minutes.

On macOS, this would cause a problem with compiled modules if you tried to use the same `node_modules` to run the app natively. However, if running via docker-compose in https://github.com/sharelatex/sharelatex-dev-environment, this is no longer a problem.
2018-01-15 09:36:41 +00:00
..
app Merge pull request #257 from sharelatex/ta-remove-proxy-headers 2018-01-12 15:48:18 +02:00
bin CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
config CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
data
modules Update modules gitignore to ignore everything except specific modules 2017-11-30 13:32:23 +00:00
public CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
test Merge pull request #251 from sharelatex/fix-group-member-subscription-page 2018-01-11 14:41:31 +00:00
.gitignore CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
.npmignore
.nvmrc Merge branch 'master' into node-6.9 2017-04-05 10:15:51 +01:00
.travis.yml
app.coffee
docker-compose.yml CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
Gruntfile.coffee libs -> libraries 2017-12-13 17:15:44 +00:00
Jenkinsfile CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
Makefile CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
Makefile.module CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
nodemon.frontend.json CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
nodemon.json CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
npm-shrinkwrap.json CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
package.json CI and local dev environment improvements 2018-01-15 09:36:41 +00:00
README.md Get module unit tests running inside Docker as well as main tests 2017-11-27 17:10:58 +00:00

web-sharelatex

web-sharelatex is the front-end web service of the open-source web-based collaborative LaTeX editor, ShareLaTeX. It serves all the HTML pages, CSS and javascript to the client. web-sharelatex also contains a lot of logic around creating and editing projects, and account management.

The rest of the ShareLaTeX stack, along with information about contributing can be found in the sharelatex/sharelatex repository.

Build process

web-sharelatex uses Grunt to build its front-end related assets.

Image processing tasks are commented out in the gruntfile and the needed packages aren't presently in the project's package.json. If the images need to be processed again (minified and sprited), start by fetching the packages (npm install grunt-contrib-imagemin grunt-sprity), then decomment the tasks in Gruntfile.coffee. After this, the tasks can be called (explicitly, via grunt imagemin and grunt sprity).

New Docker-based build process

Note that the Grunt workflow from above should still work, but we are transitioning to a Docker based testing workflow, which is documented below:

Running the app

The app runs natively using npm and Node on the local system:

$ npm install
$ npm run start

Ideally the app would run in Docker like the tests below, but with host networking not supported in OS X, we need to run it natively until all services are Dockerised.

Unit Tests

The test suites run in Docker.

Unit tests can be run in the test_unit container defined in docker-compose.tests.yml.

The makefile contains a short cut to run these:

make install # Only needs running once, or when npm packages are updated
make unit_test

During development it is often useful to only run a subset of tests, which can be configured with arguments to the mocha CLI:

make unit_test MOCHA_ARGS='--grep=AuthorizationManager'

Acceptance Tests

Acceptance tests are run against a live service, which runs in the acceptance_test container defined in docker-compose.tests.yml.

To run the tests out-of-the-box, the makefile defines:

make install # Only needs running once, or when npm packages are updated
make acceptance_test

However, during development it is often useful to leave the service running for rapid iteration on the acceptance tests. This can be done with:

make acceptance_test_start_service
make acceptance_test_run # Run as many times as needed during development
make acceptance_test_stop_service

make acceptance_test just runs these three commands in sequence.

During development it is often useful to only run a subset of tests, which can be configured with arguments to the mocha CLI:

make acceptance_test_run MOCHA_ARGS='--grep=AuthorizationManager'

Unit test status

Unit test status

License and Credits

This project is licensed under the AGPLv3 license

Stylesheets

ShareLaTeX is based on Bootstrap, which is licensed under the MIT license. All modifications (*.less files in public/stylesheets) are also licensed under the MIT license.

Artwork

Silk icon set 1.3

We gratefully acknowledge Mark James for releasing his Silk icon set under the Creative Commons Attribution 2.5 license. Some of these icons are used within ShareLaTeX inside the public/img/silk and public/brand/icons directories.

IconShock icons

We gratefully acknowledge IconShock for use of the icons in the public/img/iconshock directory found via findicons.com

Acceptance Tests

To run the Acceptance tests:

  • set allowPublicAccess to true, either in the configuration file, or by setting the environment variable SHARELATEX_ALLOW_PUBLIC_ACCESS to true
  • start the server (grunt)
  • in a separate terminal, run grunt test:acceptance