Jakob Ackermann
54e3b57772
[misc] test/unit: add missing globals that are lazy loaded
2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8
[misc] test/unit: add missing require stubs for metrics and settings
2020-02-28 12:26:22 +01:00
Simon Detheridge
8acc9ef0ae
Make S3 partSize a setting
2020-02-17 14:06:15 +00:00
Simon Detheridge
9e7eec7a19
Add acceptance test for large files
2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74
Re-fetch file to calculate md5 if etag is not in correct format
2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0
Use large upload chunks
2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc
Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
...
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1
Revert "Add Migration Persistor, to send missing file requests to a fallback persistor"
2020-02-14 14:26:33 +00:00
Simon Detheridge
93cd55fb79
Refactor persistors to use a helper for common things
2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c
Explicitly resume stream after adding listener
2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377
Remove unnecessary 'async'
2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972
Update tests to properly check for copied files
2020-02-12 13:21:53 +00:00
Simon Detheridge
9e0b378948
Remove minipass as dependency and refactor to make things clearer
2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e
Add mechanisms to transfer files with md5-based integrity checks
...
Fix error in settings and tidy up tests
Remove unused variable declaration
Remove .only from tests and update eslint rules to catch it in future
Use to catch errors more safely getting md5 hash
Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31
Add MigrationPersistor for sending 404 requests to a fallback persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775
Rename *PersistorManager to *Persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
Simon Detheridge
b642b3cf3a
Rename incorrectly-named 'getDirectorySize' method
2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3
Configure PersistorManager to use the new S3 manager for 'aws-sdk'
2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15
Cleanup and refactor S3PersistorManager to use aws-sdk only
2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60
Remove AWSSDKPersistorManager
2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55
Post-decaf cleanup of app.js
2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4
Post-decaf cleanup of app.js
2020-01-07 09:27:02 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912
Post-decaf cleanup of KeyBuilderTests
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86
Return full path in stubbed 'glob' method
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
Simon Detheridge
039bec02f7
Use directory names in acceptance tests, and add test for getting size
2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877
Use 'glob' to determine 'directory' size.
...
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394
Decaf cleanup for FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
4315824d3c
Minor cleanup of FilestoreApp acceptance-test helper
2020-01-07 09:25:04 +00:00
Simon Detheridge
a8158d6c8c
Cleanup and promisify acceptance tests. Run tests for all backends.
2020-01-07 09:25:04 +00:00
Simon Detheridge
006f84abeb
Decaf and promisify ImageOptimiser
2020-01-07 09:24:29 +00:00
Simon Detheridge
b6a1ea6a30
Cleanup and promisify FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
27aaff7843
Decaf cleanup for FileHandler and LocalFileWriter
...
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112
Cleanup SafeExec
2019-12-19 15:42:46 +00:00
Simon Detheridge
35d050a49c
Prettier fixes
2019-12-16 11:32:46 +00:00
Simon Detheridge
495424a94b
Fix metrics includes in SandboxedModule
2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495
Prettier: convert test/unit decaffeinated files to Prettier format
2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6
Rename test/unit/coffee to test/unit/js
2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files
2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1
decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS
2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff
decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js
2019-12-16 11:20:20 +00:00
Simon Detheridge
e4b1106761
Prettier: convert test/acceptance decaffeinated files to Prettier format
2019-12-16 11:19:43 +00:00
Simon Detheridge
b8e7abd25e
Rename test/acceptance/coffee to test/acceptance/js
2019-12-16 11:19:43 +00:00
decaffeinate
738e1d821e
decaffeinate: Run post-processing cleanups on FilestoreApp.coffee and 1 other file
2019-12-16 11:19:43 +00:00
decaffeinate
e6e75d4e19
decaffeinate: Convert FilestoreApp.coffee and 1 other file to JS
2019-12-16 11:19:43 +00:00
decaffeinate
1c7d1af4ca
decaffeinate: Rename FilestoreApp.coffee and 1 other file from .coffee to .js
2019-12-16 11:19:43 +00:00
Simon Detheridge
32b7f25404
Merge pull request #49 from das7pad/misc/node10plus-support
...
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
56b38af678
Explicitly wait for S3 in acceptance tests
2019-12-10 17:43:34 +00:00
Simon Detheridge
48aa141591
Add metric for s3 ingress
2019-12-09 17:06:00 +00:00
Simon Detheridge
1d1106bc67
Add metric for s3 egress
2019-12-09 17:06:00 +00:00
Eric Mc Sween
7d900b57bf
Fix createUnbufferedStream() function call
...
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.
This commit fixes that. I have tested again on S3.
2019-06-19 12:58:17 -04:00
Eric Mc Sween
f2521a29b9
Use AWS SDK for getFileStream()
...
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03
Return file size on HEAD request
...
This will be used by the file preview feature when it gets partial
content.
2019-06-17 08:23:59 -04:00
Jakob Ackermann
c59a3db4e8
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 02:47:53 +02:00
Henry Oswald
3fccf79ca8
cleanup
2019-02-05 12:50:08 +00:00
Henry Oswald
19007befb2
Merge branch 'master' into ho-docker
2019-02-01 16:26:45 +00:00
Brian Gough
9d93eee3e8
return a 404 error (instead of a 500) when copying a missing file
2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6
add enableConversions flag to disable conversions which we can't do on
...
k8
2019-01-07 15:54:24 +00:00
Christopher Hoskin
aab5a1af6a
Merge branch 'master' into csh-ho-docker
2019-01-02 09:51:35 +00:00
Brian Gough
cf12ec1154
use the aws sdk to copy files in S3PersistorManager
...
to work around problems with knox
2018-12-20 16:11:18 +00:00
Henry Oswald
95f0a02dee
stub out metrics in tests & inject metrics endpoint
2018-11-29 13:53:43 +00:00
Brian Gough
0d6de10b5c
Merge pull request #34 from sharelatex/bg-make-backends-consistent
...
make filestore backends consistent
2018-11-13 10:12:18 +00:00
Brian Gough
8744b3aa4e
serve file from disk to avoid read-after-write inconsistency
2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db
update unit tests
2018-11-07 16:51:06 +00:00
Henry Oswald
aacb54d54c
Merge branch 'master' into ho-docker
2018-09-14 17:16:18 +01:00
Henry Oswald
1e4e68c64a
fix settings test
2018-07-10 14:55:40 +01:00
Michael Mazour
c4e3f9eb02
Amend: tests for populating S3 settings from environment variable
2018-07-06 11:05:40 +01:00
Michael Mazour
cfbf0d81ba
Amend: fix params retrieval
2018-07-04 12:11:09 +01:00
Michael Mazour
feca8933f1
Add endpoint for arbitrary bucket fetch
...
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
2018-07-04 11:22:36 +01:00
Henry Oswald
8efd562eb3
fix broken acceptence tests, bad quotes in file path
2018-05-21 13:40:03 +01:00
Henry Oswald
e3514e84cd
fix up mocha migration tests
2018-05-21 11:02:53 +01:00
Henry Oswald
fda6cb0084
fix acceptence tests, add them back into jenkins, upgrade express
2018-05-21 09:37:54 +01:00
Henry Oswald
c23b700927
wip
2018-01-16 12:27:36 +00:00
Shane Kilkelly
e162c779d3
Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout
...
If optipng is sigkilled by timeout, then ignore that error
2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747
upgrade to node 6
2017-06-27 14:53:20 +01:00
Shane Kilkelly
d0bbe8beae
If optipng is sigkilled by timeout, then ignore that error
2017-06-21 15:18:36 +01:00
James Allen
be43330208
Allow convert command to be prefixed by security commands
2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b
add unit tests
2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa
remove project controller / handler
...
add unit test scaffolding
2016-03-13 16:22:14 -03:00
Xavier Trochu
3b6270236c
Replace indentation from 2 space to tabs.
2015-11-20 14:38:23 +01:00
Xavier Trochu
bfd41fdaf9
Add aws-sdk unit test.
...
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
2015-11-20 12:02:22 +01:00
Shane Kilkelly
a6fff2d4e4
Check for a large response body, rather than exact match on the first few bytes.
2015-09-08 11:50:43 +01:00
Shane Kilkelly
c6a81c94eb
Test the cacheWarm option for preview.
2015-09-08 11:47:43 +01:00
Shane Kilkelly
b8042ae771
Add an acceptance test that uses a pdf file.
2015-09-08 11:43:19 +01:00
Shane Kilkelly
297ad78b1e
Add an acceptance test to check the preview endpoint doesn't time out.
...
This endpoint needs a lot more testing.
2015-09-08 10:32:41 +01:00
Shane Kilkelly
da5a538095
Fix calls to PersistorManager.getFileStream which were missed in initial refactor.
...
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
2015-09-08 09:23:08 +01:00
Shane Kilkelly
82af1be756
Improve testing of error conditions in the FSPersistorManager.
2015-09-01 12:10:30 +01:00
