Simon Detheridge
2625e03a31
Add MigrationPersistor for sending 404 requests to a fallback persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775
Rename *PersistorManager to *Persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
Simon Detheridge
b642b3cf3a
Rename incorrectly-named 'getDirectorySize' method
2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3
Configure PersistorManager to use the new S3 manager for 'aws-sdk'
2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15
Cleanup and refactor S3PersistorManager to use aws-sdk only
2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60
Remove AWSSDKPersistorManager
2020-01-07 09:30:30 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912
Post-decaf cleanup of KeyBuilderTests
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86
Return full path in stubbed 'glob' method
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
Simon Detheridge
21ba083877
Use 'glob' to determine 'directory' size.
...
Check for filtered directory paths in all tests.
2020-01-07 09:26:00 +00:00
Simon Detheridge
ce90292394
Decaf cleanup for FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
006f84abeb
Decaf and promisify ImageOptimiser
2020-01-07 09:24:29 +00:00
Simon Detheridge
b6a1ea6a30
Cleanup and promisify FileConverter
2020-01-07 09:22:53 +00:00
Simon Detheridge
27aaff7843
Decaf cleanup for FileHandler and LocalFileWriter
...
Simplified code and tests where possible
2020-01-07 09:18:56 +00:00
Simon Detheridge
eacad77112
Cleanup SafeExec
2019-12-19 15:42:46 +00:00
Simon Detheridge
35d050a49c
Prettier fixes
2019-12-16 11:32:46 +00:00
Simon Detheridge
495424a94b
Fix metrics includes in SandboxedModule
2019-12-16 11:28:24 +00:00
Simon Detheridge
eae7d28495
Prettier: convert test/unit decaffeinated files to Prettier format
2019-12-16 11:20:29 +00:00
Simon Detheridge
ec60f778e6
Rename test/unit/coffee to test/unit/js
2019-12-16 11:20:27 +00:00
decaffeinate
ff04085bdd
decaffeinate: Run post-processing cleanups on AWSSDKPersistorManagerTests.coffee and 12 other files
2019-12-16 11:20:25 +00:00
decaffeinate
5cf7138ef1
decaffeinate: Convert AWSSDKPersistorManagerTests.coffee and 12 other files to JS
2019-12-16 11:20:22 +00:00
decaffeinate
72c83bdaff
decaffeinate: Rename AWSSDKPersistorManagerTests.coffee and 12 other files from .coffee to .js
2019-12-16 11:20:20 +00:00
Simon Detheridge
32b7f25404
Merge pull request #49 from das7pad/misc/node10plus-support
...
[misc] add support for node 10+
2019-12-16 09:39:00 +00:00
Simon Detheridge
48aa141591
Add metric for s3 ingress
2019-12-09 17:06:00 +00:00
Eric Mc Sween
7d900b57bf
Fix createUnbufferedStream() function call
...
In 49a21155f642670dfea264ac73fb60241f37cb87, I managed to incorrectly
write the `createUnbufferedStream()` function from the AWS SDK as
`getUnbufferedStream()` and to consistently use that naming in the unit
tests.
This commit fixes that. I have tested again on S3.
2019-06-19 12:58:17 -04:00
Eric Mc Sween
f2521a29b9
Use AWS SDK for getFileStream()
...
The AWS SDK has a retry strategy to deal with rate limiting or transient
unavailability of S3. We hope it will reduce our error rates.
2019-06-19 08:18:11 -04:00
Eric Mc Sween
3575c89d03
Return file size on HEAD request
...
This will be used by the file preview feature when it gets partial
content.
2019-06-17 08:23:59 -04:00
Jakob Ackermann
c59a3db4e8
[FSPersistorManager] fix the stream opening for node10+
...
Attaching a `readable` listener causes the stream to hang otherwise.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2019-05-02 02:47:53 +02:00
Henry Oswald
3fccf79ca8
cleanup
2019-02-05 12:50:08 +00:00
Henry Oswald
19007befb2
Merge branch 'master' into ho-docker
2019-02-01 16:26:45 +00:00
Brian Gough
9d93eee3e8
return a 404 error (instead of a 500) when copying a missing file
2019-01-09 10:31:59 +00:00
Henry Oswald
3614f217e6
add enableConversions flag to disable conversions which we can't do on
...
k8
2019-01-07 15:54:24 +00:00
Christopher Hoskin
aab5a1af6a
Merge branch 'master' into csh-ho-docker
2019-01-02 09:51:35 +00:00
Brian Gough
cf12ec1154
use the aws sdk to copy files in S3PersistorManager
...
to work around problems with knox
2018-12-20 16:11:18 +00:00
Henry Oswald
95f0a02dee
stub out metrics in tests & inject metrics endpoint
2018-11-29 13:53:43 +00:00
Brian Gough
0d6de10b5c
Merge pull request #34 from sharelatex/bg-make-backends-consistent
...
make filestore backends consistent
2018-11-13 10:12:18 +00:00
Brian Gough
8744b3aa4e
serve file from disk to avoid read-after-write inconsistency
2018-11-08 15:09:34 +00:00
Brian Gough
78ccf453db
update unit tests
2018-11-07 16:51:06 +00:00
Henry Oswald
aacb54d54c
Merge branch 'master' into ho-docker
2018-09-14 17:16:18 +01:00
Henry Oswald
1e4e68c64a
fix settings test
2018-07-10 14:55:40 +01:00
Michael Mazour
c4e3f9eb02
Amend: tests for populating S3 settings from environment variable
2018-07-06 11:05:40 +01:00
Michael Mazour
cfbf0d81ba
Amend: fix params retrieval
2018-07-04 12:11:09 +01:00
Michael Mazour
feca8933f1
Add endpoint for arbitrary bucket fetch
...
Add `/bucket/:bucket/key/*`, which fetches the file from the given bucket at the given path. Uses auth stored at `settings.filestore.s3.{{bucketName}}` if present, and otherwise default auth.
2018-07-04 11:22:36 +01:00
Henry Oswald
e3514e84cd
fix up mocha migration tests
2018-05-21 11:02:53 +01:00
Shane Kilkelly
e162c779d3
Merge pull request #24 from sharelatex/sk-ignore-png-optimiser-timeout
...
If optipng is sigkilled by timeout, then ignore that error
2017-08-03 09:22:43 +01:00
Shane Kilkelly
eaa72d0747
upgrade to node 6
2017-06-27 14:53:20 +01:00
Shane Kilkelly
d0bbe8beae
If optipng is sigkilled by timeout, then ignore that error
2017-06-21 15:18:36 +01:00
James Allen
be43330208
Allow convert command to be prefixed by security commands
2016-05-09 11:37:35 +01:00
Henrique Santos
ffaac4c81b
add unit tests
2016-03-13 20:45:48 -03:00
Henrique Santos
0bb12f32fa
remove project controller / handler
...
add unit test scaffolding
2016-03-13 16:22:14 -03:00
Xavier Trochu
3b6270236c
Replace indentation from 2 space to tabs.
2015-11-20 14:38:23 +01:00
Xavier Trochu
bfd41fdaf9
Add aws-sdk unit test.
...
Fix Aws-Sdk persistor to return a correct error on file not found.
Fix FileHandler after some change were lost on a previous merge
2015-11-20 12:02:22 +01:00
Shane Kilkelly
da5a538095
Fix calls to PersistorManager.getFileStream which were missed in initial refactor.
...
This should fix a breakage in the template publishing workflow.
Ref: b70b837b7ba29b0631f3105ec1dd5f6cdebbd195
2015-09-08 09:23:08 +01:00
Shane Kilkelly
82af1be756
Improve testing of error conditions in the FSPersistorManager.
2015-09-01 12:10:30 +01:00
Shane Kilkelly
8bbf81f4bc
If the S3 service produces an error, handle it.
2015-09-01 11:36:28 +01:00
Shane Kilkelly
ed980a21e1
Test the FileHandler layer.
2015-08-28 13:50:40 +01:00
Shane Kilkelly
387e96ab67
Refactor to make _get_range testable.
2015-08-28 12:02:50 +01:00
Shane Kilkelly
97cdc243d3
Parse out the range header and set start and end properties on options.
2015-08-28 11:45:16 +01:00
Shane Kilkelly
76506ed91e
Pass start/end options down to the S3PersistorManager.
...
Test that those options cause a 'Range' header to be attached to the
request sent to S3.
2015-08-28 10:09:41 +01:00
Shane Kilkelly
d88736e3b5
Pass the start and end options down to fs.createReadStream, and test for same.
2015-08-27 16:12:11 +01:00
Shane Kilkelly
57aedefdd3
Begin work towards obeying 'Range' headers, thread the opts down to persistence layer.
2015-08-27 15:21:33 +01:00
James Allen
801b5653e4
Fix unit test when false is not at /bin/false
2015-03-20 14:27:25 +00:00
Brian Gough
143d44e54b
add tests for SafeExec module
2015-03-13 10:15:53 +00:00
Brian Gough
0e5abe1ff8
update tests to use safe_exec
2015-03-13 09:31:43 +00:00
Brian Gough
8b9213d09b
fix getFileStream unit test as callback no longer fires immediately
...
success callback is now called only when file is readable
2014-12-22 13:07:30 +00:00
Brian Gough
efd6035849
test against rimraf module when deleting directory recursively
2014-12-22 13:01:39 +00:00
Brian Gough
3f18f6a714
update unit tests - FileHandler now calls deleteDirectory on cache directory
2014-12-22 12:58:41 +00:00
Henry Oswald
04bffaeceb
get local file writer tests using fake settings
2014-06-17 13:33:36 +01:00
Henry Oswald
c8c758642a
removed reference to easy image
2014-06-17 12:47:53 +01:00
Henry Oswald
2f22563d59
changed key for templates to put /v/ in for version
2014-03-05 17:39:27 +00:00
Henry Oswald
1e1c14e5fe
Merge pull request #11 from cwoac/FSPersistorManager
...
Fs persistor manager - merge from cwoac
2014-03-05 10:39:26 +00:00
Oliver Matthews
f920fd0b16
match refactor_config on cwoac/sharelatex
2014-03-04 15:24:50 +00:00
Henry Oswald
711e95a82a
delete converted file when finished to clean up
2014-03-04 14:30:52 +00:00
Oliver Matthews
88cc89a0d1
filter /s from key ids
2014-03-01 15:10:47 +00:00
Oliver Matthews
ff96f37b2d
unit test passing version of FSPersistorManager
2014-02-26 15:10:55 +00:00
Oliver Matthews
e0f0bfb0ea
revert settings.persistorManager
2014-02-26 11:49:15 +00:00
Oliver Matthews
4b5f9ad4fb
make s3 chosen if no config option specified
...
rename *wrapper to FilestoreManager
remove ToS3 from filestore commands
rename configuration option for filestore.
add details of interface to FilestoreWrapper.coffee for reference.
remove api comments
rename settings.filestoreManager to settings.persistorManager
2014-02-26 10:24:51 +00:00
Henry Oswald
c9af5e5420
Renamed fsWrapper to PersistorManager
...
not 100% happy with that still but its the best I can see so far
2014-02-26 09:15:37 +00:00
Henry Oswald
49ffeb633d
fix where cacheWarm param is read from and improved some logging
2014-02-26 09:06:47 +00:00
Oliver Matthews
5f87dfb3bf
add unit tests for fsWrapper
...
make other unit tests no longer depend on fsWrapper
2014-02-25 15:52:30 +00:00
Oliver Matthews
c43533b5cb
move test wrapper into fsWrapper where appropriate
2014-02-25 10:54:51 +00:00
Oliver Matthews
f68080d56c
create mixin to allow switching of backend
2014-02-25 09:10:22 +00:00
Henry Oswald
fada5ecaf4
converted get stream to use knox, test to see if it is more robust
2014-02-19 20:56:45 +00:00
Henry Oswald
d0600a4b64
changed optimisation to optipng and added timeouts to conversions
2014-02-19 14:48:15 +00:00
Henry Oswald
f99a6dc39b
changed file converted to use child process with nice
2014-02-19 14:48:12 +00:00
James Allen
3199aad601
Initial open source commit
2014-02-14 16:39:05 +00:00
