Simon Detheridge
4e507197fc
Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
...
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1
Revert "Add Migration Persistor, to send missing file requests to a fallback persistor"
2020-02-14 14:26:33 +00:00
Simon Detheridge
93cd55fb79
Refactor persistors to use a helper for common things
2020-02-12 13:27:12 +00:00
Simon Detheridge
9e0b378948
Remove minipass as dependency and refactor to make things clearer
2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e
Add mechanisms to transfer files with md5-based integrity checks
...
Fix error in settings and tidy up tests
Remove unused variable declaration
Remove .only from tests and update eslint rules to catch it in future
Use to catch errors more safely getting md5 hash
Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31
Add MigrationPersistor for sending 404 requests to a fallback persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775
Rename *PersistorManager to *Persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
Simon Detheridge
b642b3cf3a
Rename incorrectly-named 'getDirectorySize' method
2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3
Configure PersistorManager to use the new S3 manager for 'aws-sdk'
2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15
Cleanup and refactor S3PersistorManager to use aws-sdk only
2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60
Remove AWSSDKPersistorManager
2020-01-07 09:30:30 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912
Post-decaf cleanup of KeyBuilderTests
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86
Return full path in stubbed 'glob' method
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877
Use 'glob' to determine 'directory' size.
...
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394
Decaf cleanup for FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
006f84abeb
Decaf and promisify ImageOptimiser
2020-01-07 09:24:29 +00:00
Simon Detheridge
b6a1ea6a30
Cleanup and promisify FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
27aaff7843
Decaf cleanup for FileHandler and LocalFileWriter
...
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112
Cleanup SafeExec
2019-12-19 15:42:46 +00:00
Simon Detheridge
35d050a49c
Prettier fixes
2019-12-16 11:32:46 +00:00
Simon Detheridge
495424a94b
Fix metrics includes in SandboxedModule
2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495
Prettier: convert test/unit decaffeinated files to Prettier format
2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6
Rename test/unit/coffee to test/unit/js
2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files
2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1
decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS
2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff
decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js
2019-12-16 11:20:20 +00:00
Simon Detheridge
32b7f25404
Merge pull request #49 from das7pad/misc/node10plus-support
...
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
48aa141591
Add metric for s3 ingress
2019-12-09 17:06:00 +00:00
Eric Mc Sween
7d900b57bf
Fix createUnbufferedStream() function call
...
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.
This commit fixes that. I have tested again on S3.
2019-06-19 12:58:17 -04:00
Eric Mc Sween
f2521a29b9
Use AWS SDK for getFileStream()
...
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03
Return file size on HEAD request
...
This will be used by the file preview feature when it gets partial
content.
2019-06-17 08:23:59 -04:00
Jakob Ackermann
c59a3db4e8
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 02:47:53 +02:00
Henry Oswald
3fccf79ca8
cleanup
2019-02-05 12:50:08 +00:00
Henry Oswald
19007befb2
Merge branch 'master' into ho-docker
2019-02-01 16:26:45 +00:00
Brian Gough
9d93eee3e8
return a 404 error (instead of a 500) when copying a missing file
2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6
add enableConversions flag to disable conversions which we can't do on
...
k8
2019-01-07 15:54:24 +00:00
Christopher Hoskin
aab5a1af6a
Merge branch 'master' into csh-ho-docker
2019-01-02 09:51:35 +00:00
Brian Gough
cf12ec1154
use the aws sdk to copy files in S3PersistorManager
...
to work around problems with knox
2018-12-20 16:11:18 +00:00
Henry Oswald
95f0a02dee
stub out metrics in tests & inject metrics endpoint
2018-11-29 13:53:43 +00:00
Brian Gough
0d6de10b5c
Merge pull request #34 from sharelatex/bg-make-backends-consistent
...
make filestore backends consistent
2018-11-13 10:12:18 +00:00
Brian Gough
8744b3aa4e
serve file from disk to avoid read-after-write inconsistency
2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db
update unit tests
2018-11-07 16:51:06 +00:00
Henry Oswald
aacb54d54c
Merge branch 'master' into ho-docker
2018-09-14 17:16:18 +01:00
Henry Oswald
1e4e68c64a
fix settings test
2018-07-10 14:55:40 +01:00
Michael Mazour
c4e3f9eb02
Amend: tests for populating S3 settings from environment variable
2018-07-06 11:05:40 +01:00
Michael Mazour
cfbf0d81ba
Amend: fix params retrieval
2018-07-04 12:11:09 +01:00
Michael Mazour
feca8933f1
Add endpoint for arbitrary bucket fetch
...
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
2018-07-04 11:22:36 +01:00
Henry Oswald
e3514e84cd
fix up mocha migration tests
2018-05-21 11:02:53 +01:00
Shane Kilkelly
e162c779d3
Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout
...
If optipng is sigkilled by timeout, then ignore that error
2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747
upgrade to node 6
2017-06-27 14:53:20 +01:00
Shane Kilkelly
d0bbe8beae
If optipng is sigkilled by timeout, then ignore that error
2017-06-21 15:18:36 +01:00
James Allen
be43330208
Allow convert command to be prefixed by security commands
2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b
add unit tests
2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa
remove project controller / handler
...
add unit test scaffolding
2016-03-13 16:22:14 -03:00
Xavier Trochu
3b6270236c
Replace indentation from 2 space to tabs.
2015-11-20 14:38:23 +01:00
Xavier Trochu
bfd41fdaf9
Add aws-sdk unit test.
...
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
2015-11-20 12:02:22 +01:00
Shane Kilkelly
da5a538095
Fix calls to PersistorManager.getFileStream which were missed in initial refactor.
...
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
2015-09-08 09:23:08 +01:00
Shane Kilkelly
82af1be756
Improve testing of error conditions in the FSPersistorManager.
2015-09-01 12:10:30 +01:00
Shane Kilkelly
8bbf81f4bc
If the S3 service produces an error, handle it.
2015-09-01 11:36:28 +01:00
Shane Kilkelly
ed980a21e1
Test the FileHandler layer.
2015-08-28 13:50:40 +01:00
Shane Kilkelly
387e96ab67
Refactor to make _get_range testable.
2015-08-28 12:02:50 +01:00
Shane Kilkelly
97cdc243d3
Parse out the range header and set start and end properties on options.
2015-08-28 11:45:16 +01:00
Shane Kilkelly
76506ed91e
Pass start/end options down to the S3PersistorManager.
...
Test that those options cause a 'Range' header to be attached to the
request sent to S3.
2015-08-28 10:09:41 +01:00
Shane Kilkelly
d88736e3b5
Pass the start and end options down to fs.createReadStream, and test for same.
2015-08-27 16:12:11 +01:00
Shane Kilkelly
57aedefdd3
Begin work towards obeying 'Range' headers, thread the opts down to persistence layer.
2015-08-27 15:21:33 +01:00
James Allen
801b5653e4
Fix unit test when false is not at /bin/false
2015-03-20 14:27:25 +00:00
Brian Gough
143d44e54b
add tests for SafeExec module
2015-03-13 10:15:53 +00:00
Brian Gough
0e5abe1ff8
update tests to use safe_exec
2015-03-13 09:31:43 +00:00
Brian Gough
8b9213d09b
fix getFileStream unit test as callback no longer fires immediately
...
success callback is now called only when file is readable
2014-12-22 13:07:30 +00:00
Brian Gough
efd6035849
test against rimraf module when deleting directory recursively
2014-12-22 13:01:39 +00:00
Brian Gough
3f18f6a714
update unit tests - FileHandler now calls deleteDirectory on cache directory
2014-12-22 12:58:41 +00:00
Henry Oswald
04bffaeceb
get local file writer tests using fake settings
2014-06-17 13:33:36 +01:00
Henry Oswald
c8c758642a
removed reference to easy image
2014-06-17 12:47:53 +01:00
Henry Oswald
2f22563d59
changed key for templates to put /v/ in for version
2014-03-05 17:39:27 +00:00
Henry Oswald
1e1c14e5fe
Merge pull request #11 from cwoac/FSPersistorManager
...
Fs persistor manager - merge from cwoac
2014-03-05 10:39:26 +00:00
Oliver Matthews
f920fd0b16
match refactor_config on cwoac/sharelatex
2014-03-04 15:24:50 +00:00
Henry Oswald
711e95a82a
delete converted file when finished to clean up
2014-03-04 14:30:52 +00:00
Oliver Matthews
88cc89a0d1
filter /s from key ids
2014-03-01 15:10:47 +00:00
Oliver Matthews
ff96f37b2d
unit test passing version of FSPersistorManager
2014-02-26 15:10:55 +00:00
Oliver Matthews
e0f0bfb0ea
revert settings.persistorManager
2014-02-26 11:49:15 +00:00
Oliver Matthews
4b5f9ad4fb
make s3 chosen if no config option specified
...
rename *wrapper to FilestoreManager
remove ToS3 from filestore commands
rename configuration option for filestore.
add details of interface to FilestoreWrapper.coffee for reference.
remove api comments
rename settings.filestoreManager to settings.persistorManager
2014-02-26 10:24:51 +00:00
Henry Oswald
c9af5e5420
Renamed fsWrapper to PersistorManager
...
not 100% happy with that still but its the best I can see so far
2014-02-26 09:15:37 +00:00
Henry Oswald
49ffeb633d
fix where cacheWarm param is read from and improved some logging
2014-02-26 09:06:47 +00:00
Oliver Matthews
5f87dfb3bf
add unit tests for fsWrapper
...
make other unit tests no longer depend on fsWrapper
2014-02-25 15:52:30 +00:00
Oliver Matthews
c43533b5cb
move test wrapper into fsWrapper where appropriate
2014-02-25 10:54:51 +00:00
Oliver Matthews
f68080d56c
create mixin to allow switching of backend
2014-02-25 09:10:22 +00:00
Henry Oswald
fada5ecaf4
converted get stream to use knox, test to see if it is more robust
2014-02-19 20:56:45 +00:00
Henry Oswald
d0600a4b64
changed optimisation to optipng and added timeouts to conversions
2014-02-19 14:48:15 +00:00
