Commit graph

484 commits

Author SHA1 Message Date
Simon Detheridge
6dcf351377 Remove unnecessary 'async' 2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972 Update tests to properly check for copied files 2020-02-12 13:21:53 +00:00
Simon Detheridge
42a5d168df Remove unused packages 2020-02-12 13:21:52 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775 Rename *PersistorManager to *Persistor 2020-02-12 13:19:48 +00:00
Brian Gough
a2e1591e93 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Simon Detheridge
e3613e52db Fix log info assignment in error handler 2020-02-12 10:34:56 +00:00
Jakob Ackermann
1341af04f5 [misc] update the build scripts to 1.3.5 2020-02-11 18:58:55 +01:00
Jakob Ackermann
ca94225c19 Merge pull request #80 from overleaf/jpa-cleanup-install-deps
[misc] cleanup install deps
2020-02-10 17:16:44 +01:00
Brian Gough
14ba1ea1c1 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
ac4361d560 [misc] dockerignore: hide the local data dirs 2020-01-31 15:50:53 +01:00
Jakob Ackermann
c1d81dc1ea [misc] install a maintained ghostscript version and delete package lists
also do not install vim and bail out in case any command exited with a
 non zero exit code.
2020-01-31 15:50:53 +01:00
Simon Detheridge
997434c930 Remove redundant 'res.logInfo' 2020-01-14 20:55:33 +00:00
Simon Detheridge
035364b62e Fix RequestLogger object lifecycle 2020-01-14 17:15:01 +00:00
Simon Detheridge
28fb998719 Refactor request logger into a class 2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f Improve logging middleware to add info methods to request object 2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa Simplify logging to log only once per http request 2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c Handle AccessDenied and stream-premature-close errors
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59 Use correct object path for s3BucketCreds 2020-01-08 11:32:05 +00:00
Simon Detheridge
3bf51cac67 Improve naming on internal '_client' method and use Map over object 2020-01-08 09:17:30 +00:00
Simon Detheridge
80d41cf51b Move bucket-specific file endpoint into FileController 2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13 Re-add bucket-specific credentials support for S3 2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3 Clean up settings tests 2020-01-07 09:46:53 +00:00
Simon Detheridge
ffd002dc76 Add explanatory comment 2020-01-07 09:30:30 +00:00
Simon Detheridge
b642b3cf3a Rename incorrectly-named 'getDirectorySize' method 2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3 Configure PersistorManager to use the new S3 manager for 'aws-sdk' 2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15 Cleanup and refactor S3PersistorManager to use aws-sdk only 2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60 Remove AWSSDKPersistorManager 2020-01-07 09:30:30 +00:00
Simon Detheridge
e27cf4db7b Pass pipeline errors onto 'next' error handler 2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55 Post-decaf cleanup of app.js 2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:27:02 +00:00
Simon Detheridge
6a679023d3 Fix order of .status().send() 2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4 Post-decaf cleanup of app.js 2020-01-07 09:27:02 +00:00
Simon Detheridge
0329c759dc Remove wrapper function in PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
dbfacce988 Post-decaf cleanup of PersistorManager 2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912 Post-decaf cleanup of KeyBuilderTests 2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a Decaf cleanup of FileController
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86 Return full path in stubbed 'glob' method 2020-01-07 09:26:01 +00:00
Simon Detheridge
fccb288985 Add comment regarding deleteDirectory being internal-only 2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69 Throw errors more consistently in FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4 Delete temporary file when error in writing to stream 2020-01-07 09:26:00 +00:00
Simon Detheridge
039bec02f7 Use directory names in acceptance tests, and add test for getting size 2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877 Use 'glob' to determine 'directory' size.
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394 Decaf cleanup for FSPersistorManager 2020-01-07 09:26:00 +00:00
Simon Detheridge
4315824d3c Minor cleanup of FilestoreApp acceptance-test helper 2020-01-07 09:25:04 +00:00
Simon Detheridge
a8158d6c8c Cleanup and promisify acceptance tests. Run tests for all backends. 2020-01-07 09:25:04 +00:00
Simon Detheridge
006f84abeb Decaf and promisify ImageOptimiser 2020-01-07 09:24:29 +00:00
Simon Detheridge
42adc59d01 Clean up and promisify health-check controller and KeyBuilder 2020-01-07 09:23:41 +00:00
Simon Detheridge
ac2d05ecb3 Remove unnecessary logging from FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
b6a1ea6a30 Cleanup and promisify FileConverter 2020-01-07 09:22:53 +00:00
Simon Detheridge
a216be5fd3 Improve error handling in FileHandler 2020-01-07 09:18:59 +00:00
Simon Detheridge
27aaff7843 Decaf cleanup for FileHandler and LocalFileWriter
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112 Cleanup SafeExec 2019-12-19 15:42:46 +00:00
Simon Detheridge
f19d82369d Bump eslintrc up again to last-known-compatible version with eslit-prettier 2019-12-18 09:52:18 +00:00
Simon Detheridge
b94df9952b Add OError 2019-12-18 09:34:25 +00:00
Simon Detheridge
42f6d10031 Remove mongo and redis from dependencies 2019-12-17 15:19:19 +00:00
Simon Detheridge
0b3b71b857 Fall back to old AWS env vars if present 2019-12-17 09:57:51 +00:00
Simon Detheridge
b7e098208f Update build files from build_scripts 2019-12-16 17:17:25 +00:00
Simon Detheridge
29f333d60c Placate eslint 2019-12-16 15:38:00 +00:00
Simon Detheridge
7870c8185e Add linting to makefile 2019-12-16 11:58:22 +00:00
Simon Detheridge
230baef8fa Add linting tasks to package.json 2019-12-16 11:32:58 +00:00
Simon Detheridge
35d050a49c Prettier fixes 2019-12-16 11:32:46 +00:00
Simon Detheridge
6974fd640d Remove unusued cluster.coffee 2019-12-16 11:29:25 +00:00
Simon Detheridge
495424a94b Fix metrics includes in SandboxedModule 2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495 Prettier: convert test/unit decaffeinated files to Prettier format 2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6 Rename test/unit/coffee to test/unit/js 2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files 2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1 decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS 2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js 2019-12-16 11:20:20 +00:00
Simon Detheridge
e06c03e536 Fix decaf error in FSPersistorManager 2019-12-16 11:19:56 +00:00
Simon Detheridge
0b6e725137 Remove coffeescript compiles from package.json 2019-12-16 11:19:56 +00:00
Simon Detheridge
9943e2cc8f Prettier: convert individual decaffeinated files to Prettier format 2019-12-16 11:19:56 +00:00
Simon Detheridge
1d295ef7e4 Decaffeinate: convert individual files to js 2019-12-16 11:19:55 +00:00
Simon Detheridge
d70a33c4f5 Rename individual coffee files to js files 2019-12-16 11:19:43 +00:00
Simon Detheridge
e4b1106761 Prettier: convert test/acceptance decaffeinated files to Prettier format 2019-12-16 11:19:43 +00:00
Simon Detheridge
b8e7abd25e Rename test/acceptance/coffee to test/acceptance/js 2019-12-16 11:19:43 +00:00
decaffeinate
738e1d821e decaffeinate: Run post-processing cleanups on FilestoreApp.coffee and 1 other file 2019-12-16 11:19:43 +00:00
decaffeinate
e6e75d4e19 decaffeinate: Convert FilestoreApp.coffee and 1 other file to JS 2019-12-16 11:19:43 +00:00
decaffeinate
1c7d1af4ca decaffeinate: Rename FilestoreApp.coffee and 1 other file from .coffee to .js 2019-12-16 11:19:43 +00:00
Simon Detheridge
6225f2f236 Prettier: convert app/js decaffeinated files to Prettier format 2019-12-16 11:19:36 +00:00
Simon Detheridge
601861ee58 Remove js from dockerignore 2019-12-16 10:54:12 +00:00
Simon Detheridge
3718c90c7a Rename app/coffee dir to app/js 2019-12-16 10:54:08 +00:00
decaffeinate
fcbb13b5bf decaffeinate: Run post-processing cleanups on AWSSDKPersistorManager.coffee and 13 other files 2019-12-16 10:54:06 +00:00
decaffeinate
6bd8452f19 decaffeinate: Convert AWSSDKPersistorManager.coffee and 13 other files to JS 2019-12-16 10:54:03 +00:00
decaffeinate
8d2c87420e decaffeinate: Rename AWSSDKPersistorManager.coffee and 13 other files from .coffee to .js 2019-12-16 10:53:59 +00:00
Simon Detheridge
ae3d25c969 Decaffeinate: add eslint and prettier rc files 2019-12-16 10:53:56 +00:00
Simon Detheridge
1798efd4be Install eslint and prettier 2019-12-16 10:53:52 +00:00
Simon Detheridge
c9106eff95 Upgrade Docker image to node 10 2019-12-16 10:53:42 +00:00
Jakob Ackermann
ed97fcfcd0 [logging] do not overwrite the logger name
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:34 +00:00
Jakob Ackermann
c5e1584fcc [FSPersistorManager] fix the stream opening for node10+
Attaching a `readable` listener causes the stream to hang otherwise.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-12-16 10:53:23 +00:00
Simon Detheridge
dab44f6fec Merge branch 'master' into logging/keep-logger-name 2019-12-16 09:42:18 +00:00
Simon Detheridge
32b7f25404 Merge pull request #49 from das7pad/misc/node10plus-support
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
56b38af678 Explicitly wait for S3 in acceptance tests 2019-12-10 17:43:34 +00:00
Simon Detheridge
237c4113cd Ensure fakes3 is healthy before running tests 2019-12-10 16:36:33 +00:00
Simon Detheridge
cf684dcd98 Fix fakes3 configuration in ci yml 2019-12-10 11:38:19 +00:00
Simon Detheridge
6f326d5650 Use SSL setting based on url protocol 2019-12-09 17:41:20 +00:00