Michael Walker
ced34d91d4
Delete temporary files created by Tar on close
...
The SwapJob creates temporary files indirectly through Tar.bz2.zip.
These files are deleted with File.deleteOnExit, but there is no other
deletion mechanism. This means that temporary files will build up as
the git bridge runs, only being deleted when the JVM finally
terminates.
Instead, we can delete these temporary files as soon as they are no
longer needed. The files are wrapped in a FileInputStream and not
directly exposed, so we can simply delete the underlying file when the
stream is closed, as then the file is inaccessible.
2017-07-14 12:08:50 +01:00
Winston Li
21b769170d
Minor name change to keep swap job / s3 decoupled
2017-04-04 13:09:43 +01:00
Winston Li
5dcc4128b4
Remove sleep and reset jgit version
2017-02-17 15:57:09 +00:00
Winston Li
ee61d72e2e
Implementation and tests of GC, GcJob, S3 files
2017-02-17 13:44:11 +00:00
Winston Li
8a8d308365
Remove old printing from failing test
2017-02-16 09:16:51 +00:00
Winston Li
f29c7395d2
Tests: fix config for git init and factor all git commands into method calls
2017-01-05 14:34:03 +00:00
Winston Li
034c064c25
Fix for zip unzip test
2017-01-05 13:21:15 +00:00
Winston Li
af49325b88
Make submodule test more verbose
2017-01-04 22:35:48 +00:00
Winston Li
9c63f06d4e
Change InvalidGitRepository message
2016-12-22 13:40:12 +00:00
Winston Li
6d563ed40e
Better javadoc, improve handling of submodules
2016-12-19 12:56:58 +00:00
Winston Li
76eb0a3200
Improve javadoc and logging
2016-12-17 16:09:01 +00:00
Winston Li
d3fd17aab5
Improve javadocs around swapping/repos
2016-12-17 15:20:08 +00:00
Winston Li
25a988daa3
Add JavaDoc, minor refactor
2016-10-09 21:13:11 +01:00
Winston Li
c23c11973a
Add test confirming the dangling meta char exception is not thrown anymore ( fix #1025 )
2016-10-09 08:47:57 +01:00
Winston Li
d3eb737120
Ignore .gitignore files on git.overleaf.com repos ( fix #1281 )
2016-10-08 16:12:20 +01:00
Winston Li
a8bad0539c
Create/delete larger files in df test
2016-09-19 10:07:46 +01:00
Winston Li
fe058e8695
Use df equivalent as default for calculating used space
2016-09-19 10:01:07 +01:00
Winston Li
9a0f18e516
Increase postback timeout to 60s
2016-09-19 09:34:45 +01:00
Winston Li
e9ffcfe19e
Remove extra comma
2016-09-12 15:42:18 +01:00
Winston Li
00d512725a
Fix "evicated" typo
2016-09-12 15:16:20 +01:00
Winston Li
9b78138314
Remove duplicate call to create the last_accessed index
2016-09-12 15:13:55 +01:00
Winston Li
a8c743cb9c
Schedule next swap job after previous job finishes
2016-09-12 15:11:56 +01:00
Winston Li
d6f3682df4
Update README about running tests
2016-09-12 14:27:00 +01:00
Winston Li
d225b8cbd9
Add New Relic stuff
2016-09-06 09:13:34 +01:00
Winston Li
95a7cb2b8b
Add check for last_accessed column in case adding failed due to an error rather than because it already exists
2016-09-06 09:13:20 +01:00
Winston Li
5c878ccc70
Put files in directories that weren't commited
2016-09-06 09:13:09 +01:00
Winston Li
ea803b53d8
Remove swap table and put last_accessed column in projects
2016-09-06 09:11:53 +01:00
Winston Li
2cdf254830
README formatting
2016-09-06 09:11:32 +01:00
Winston Li
a7ba5ef59b
Ignore .idea and *.iml again.
2016-09-06 09:11:32 +01:00
Winston Li
523ea5139e
Update example conf and README
2016-09-06 09:11:32 +01:00
Winston Li
eb77d166e2
Add checkDB to upgrade the DB
2016-09-06 09:11:32 +01:00
Winston Li
52e5d2921f
Sanitise s3 info and fix for s3
2016-09-06 09:11:32 +01:00
Winston Li
9936fbe3c9
Implement and test the swap job, and add integration test
2016-09-06 09:11:32 +01:00
Winston Li
dd5694104d
Delete the SnapshotRepositoryBuilder and WLBridgedProject abominations. Set accessed time after update and push.
2016-09-06 09:11:32 +01:00
Winston Li
c459cd57af
Implement and test SwapJobImpl
2016-09-06 09:11:32 +01:00
Winston Li
a595acd0a6
Add and test GetNumProjects
2016-09-06 09:11:32 +01:00
Winston Li
4b014826d3
Add and test InMemorySwapStore
2016-09-06 09:11:32 +01:00
Winston Li
f036ff2c8b
Add and test an SQL update for only setting a project's last_accessed if it's missing
2016-09-06 09:11:32 +01:00
Winston Li
9d19cdd261
Implement and test DBStore swap_table functionality
2016-09-06 09:11:32 +01:00
Winston Li
d2b628d904
Move stuff to bridge package
2016-09-06 09:11:32 +01:00
Winston Li
f3eb32e2ec
Catch more exceptions in db init
2016-09-06 09:11:32 +01:00
Winston Li
cfc02bbcc8
Remove DB fluff layer
2016-09-06 09:11:32 +01:00
Winston Li
c3609803c0
Change DBStore to target a file
2016-09-06 09:11:32 +01:00
Winston Li
25fea8ef58
Implement bz2 support and test
2016-09-06 09:11:32 +01:00
Winston Li
8c0937511e
Add implementations, implement S3SwapStore (with only tars), FSRepoStore, Tar and File utils, add tests
2016-09-06 09:11:32 +01:00
Winston Li
692b979098
Large refactor of parts into distinct components / interfaces
2016-09-05 15:45:57 +01:00
Winston Li
5b810b64ba
Decouple jgit's Repository class from the Bridge
2016-09-05 15:45:57 +01:00
Winston Li
598ea8a6fb
Ugrade jgit and jgit.http.server to 4.4.1.201607150455-r and fix tests
2016-08-20 13:22:07 +01:00
Winston Li
3389615ef6
Use better error message for problems building repo ( #1576 )
2016-08-20 12:15:34 +01:00
Marc Egea i Sala
d143baf489
Add more logging to commit action ( #16 )
2016-06-09 10:26:22 +01:00
Marc Egea i Sala
cb2e12d3b0
Merge pull request #15 from overleaf/handler-refactor
...
Put API handlers into their own namespace
2016-06-07 07:57:07 +01:00
John Lees-Miller
4fb9cbdf19
Restore a removed logging statement
2016-06-03 16:45:16 +01:00
John Lees-Miller
7cb82f3c18
Add tests for file content API
2016-06-03 16:22:10 +01:00
John Lees-Miller
42d1236063
Check for missing project before checking key
2016-06-03 16:11:31 +01:00
John Lees-Miller
1ac07526dc
Reorganise postbacks and files under /api
2016-06-03 16:11:02 +01:00
Marc Egea i Sala
2a87513603
Merge pull request #14 from overleaf/performance-improvements
...
Build RepositoryObjectTreeWalker only once per commit
2016-06-03 15:46:32 +01:00
Marc Egea i Sala
f4d4ce0da1
Build RepositoryObjectTreeWalker only once per commit
2016-06-03 15:40:48 +01:00
John Lees-Miller
7ed93bc2cc
Log to temp file when running tests
2016-06-03 11:47:25 +01:00
John Lees-Miller
64b49904e4
Use logback so we can use two log streams
...
We want all logs on stdout, for debugging, and warnings and errors
on stderr, for remote monitoring. The SLF4J SimpleLogger can't seem
to do this, but logback does, and we can still use the same SLF4J
interface.
2016-06-03 11:47:25 +01:00
John Lees-Miller
758bc3eb8c
Add missing newline at end of file
2016-06-02 12:59:23 +01:00
John Lees-Miller
9c63ea6c93
Upgrade some testing dependencies
2016-06-02 12:41:24 +01:00
John Lees-Miller
13a75fdb95
Upgrade google http and oauth clients to 1.22.0
2016-06-02 12:26:58 +01:00
John Lees-Miller
c3725e3246
Upgrade async-http-client to 1.9.38
2016-06-02 12:22:14 +01:00
John Lees-Miller
dbaa78928e
Upgrade sqlite-jdbc to 3.8.11.2
2016-06-02 12:13:17 +01:00
John Lees-Miller
bbde2d31d8
Upgrade gson to 2.6.2
2016-06-02 12:13:13 +01:00
John Lees-Miller
ad81e7a2ee
Upgrade joda-time to 2.9.4
2016-06-02 12:01:35 +01:00
John Lees-Miller
1edaf5fe77
Upgrade to jetty 9.3.9
2016-06-02 11:57:04 +01:00
John Lees-Miller
465db49a22
Remove installation scripts
...
We now handle production deployment using some ansible playbooks based
on these scripts, so rather than having slightly different versions in
two places, I think it’s safer to remove them for now.
2016-06-02 10:39:21 +01:00
John Lees-Miller
8ea3ed5dbe
Add IDEA project files
...
Using this .gitignore file to decide what to commit:
https://github.com/github/gitignore/blob/master/Global/JetBrains.gitigno
re
2016-06-02 10:15:14 +01:00
Marc Egea i Sala
6a57dd2d9b
Improve indentation
2016-05-23 16:16:40 +01:00
Marc Egea i Sala
45254bf34a
Add minimal git config to repos
...
The test don't pass unless a minimal configuration for Git is provided.
With this change, we don't rely on the final user to provide this
configuration.
Every time a repository is cloned on tests we're adding a configuration
only for that repository that's not going to clash with any other
configuration already set.
2016-05-23 15:59:24 +01:00
Winston Li
d582e3e1cf
Remove and replace Util.sout and .serr
2016-04-18 11:03:55 +01:00
Winston Li
a409a15f07
Fix #1384 , possibly #1174 too
2016-04-05 20:39:36 +01:00
Winston Li
bb619bfaf1
Add tests that reproduce #1384 .
2016-04-05 04:17:15 +01:00
Winston Li
08f3497a9e
Improve logging around main operations.
...
PostbackManager.java: check for actual postback promise before waiting.
2016-04-03 12:17:06 +01:00
Winston Li
ab09d044a8
Improve logging for catching #1174 , and fix tests.
2016-04-01 05:23:59 +01:00
Winston Li
a4df78c73d
Improve logging.
...
- Add sanitised Config and test
- Fix issue with file permissions
2016-03-31 06:35:09 +01:00
Winston Li
559b32a22f
Integrate slf4j simple logger and change some messages. Declare some exceptions in tests
2016-03-11 09:46:03 +00:00
Marc Egea i Sala
7048032463
Exception to handle files too big to be processed
2016-02-04 10:22:18 +00:00
Marc Egea i Sala
b3e140ed40
Provide custom message on 401 errors
2016-01-19 10:44:03 +00:00
Winston Li
850cbcf0c6
Merge bug fixes with oauth2 skipping
2016-01-04 04:34:26 +00:00
Winston Li
53d3033b5c
Implement auth skipping on non-protected projects
2016-01-04 04:14:50 +00:00
Winston Li
a9a74890d2
Delete stuff that should be gitignored
2016-01-04 02:53:40 +00:00
Marc Egea i Sala
866f7dedee
Fix error log folder
...
Error log is stored under /var/log/err.log. This PR fixes it and stores the log under /var/log/wlgb/err.log
2015-11-30 11:06:00 +00:00
Marc Egea i Sala
dfb4835cc1
Merge pull request #6 from overleaf/version-update
...
Update local version after processing snapshots
2015-11-30 11:00:41 +00:00
John Lees-Miller
318e11deeb
mock the persistent store with jmock
2015-11-24 12:06:43 +00:00
John Lees-Miller
e2574ab163
fix whitespace
2015-11-24 12:06:05 +00:00
John Lees-Miller
62749a0dab
update to IDEA 15
2015-11-24 12:05:54 +00:00
John Lees-Miller
624c3e6af1
update to junit 4.12 to get jmock working
...
Also moved junit to the test scope.
2015-11-24 12:05:24 +00:00
Marc Egea i Sala
937d9c82b4
Update local version after processing snapshots
2015-11-23 16:24:22 +00:00
Marc Egea i Sala
6a25132f0c
InteliJ IDEA project library files
2015-11-23 12:30:46 +00:00
Marc Egea i Sala
229ed1c09d
Try to download file if it's not in the git tree
2015-11-23 09:36:24 +00:00
Marc Egea i Sala
18bb534f96
Handle 401 errors as ptorected projects
2015-11-19 10:40:07 +00:00
Winston Li
0156aa8c51
Update README.md
2015-10-25 13:58:40 +00:00
Winston Li
1872eba223
Implementation of OAuth2
2015-10-25 13:44:38 +00:00
Winston Li
c3db415ce7
Implement config changes and tests
2015-10-25 11:19:02 +00:00
Marc Egea i Sala
6ae47e8821
Bugfix: using statusElement and not codeElement again
2015-10-02 15:22:55 +01:00
Marc Egea i Sala
3970fb146c
Handle response errors through Util.getCodeFromResponse
...
Centralizing the exceptions to provide more details on fail.
2015-09-30 10:45:58 +01:00
Marc Egea i Sala
8648f8bde7
Use WLUser to provide default name and email
2015-09-29 11:36:10 +01:00
Marc Egea i Sala
61a5c5870c
Support for anonymous saved versions
...
This fixes a case when de API sends a version without user. There was a
bug allowing to create anonymous versions in the application and we
have to support the old data.
The problem here is that SnapshotInfo classes are inflated from json
via Gson. This method does not call any constructor and, since the json
does not include the ‘user’ key, the bridge crashes because we’re not
expecting null users.
I’m not happy with this fix, but is the minimum solution that does not
affect anything else.
2015-09-25 16:15:37 +01:00
