Brian Gough
65cf4cf7c7
make flush to track-changes failsafe
2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8
remove console.log from tests
2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73
fix missing bodyParser limit
2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb
Bump request from 2.25.0 to 2.47.0
...
Bumps [request](https://github.com/request/request ) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases )
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md )
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da
Bump express from 3.3.4 to 3.11.0
...
Bumps [express](https://github.com/expressjs/express ) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases )
- [Changelog](https://github.com/expressjs/express/blob/master/History.md )
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be
bump build script to 1.1.24
2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e
replace private link with public one
2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece
Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest
...
add ioredis maxRetriesPerRequest
2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847
set maxRetriesPerRequest from environment variable
2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4
Merge pull request #95 from overleaf/bg-update-ioredis
...
update to ioredis 4.14.1
2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd
add ioredis option maxRetriesPerRequest: 0
...
see https://github.com/luin/ioredis/issues/965
2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c
remove old unused ioredis keepalive option
2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224
update to ioredis 4.14.1
2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b
logs out when a redis event occurs
2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76
avoid project history queues building up with deferred flush
2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac
Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes
...
add continuous background flush
2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8
add comment about background flush limit
2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6
start background flush after http server has started
2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70
add smoothing of delete spikes
2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052
add random jitter to cutoff time
2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4
make background flush more adaptive
2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186
add continuous background flush
2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1
fix getDocTimestamps for multiple docs
2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe
Merge pull request #89 from overleaf/bg-queue-deletes
...
queue deletes for deferred processing
2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33
Update config
2019-09-28 11:07:33 +01:00
Brian Gough
260923f291
keep flushQueuedProjects in the foreground
2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660
check timestamps array length
2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43
allow flush to complete in acceptance test
2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af
remove unused dryRun option
...
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b
fix log line
2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a
fix error logging
2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa
run flush of queued projects in the background
2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6
add comment about ZPOPMIN
2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa
for simplicity keep the cutoff time the same while flushing the queue
2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca
remove unnecessary check
2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f
add an acceptance test for flush with queue processing
2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9
add comments
2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809
add metric for queue length
2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb
upgrade redis-sharelatex to 1.0.10
2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059
added log lines for all project flushing
2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d
add logger into project flusher
2019-08-29 20:36:00 +01:00
Brian Gough
e75f2cc325
add unit tests for skipping history flush
2019-08-15 11:01:45 +01:00
Brian Gough
7493462154
add acceptance test for realtime shutdown
2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88
skip history flush when project is cleared by realtime shutdown
...
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05
Merge pull request #79 from overleaf/ho-mongo-connection-string
...
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7
Merge pull request #82 from overleaf/bg-mongo-health-check
...
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4
Improve/fix serializers for update logging ( #80 )
...
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19
add a combined health check for mongo and redis
2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-07 15:44:57 +01:00
Simon Detheridge
df9ca8b272
Add serializer to print only length of large fields in production
2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d
remove unnecessary check for doc_id
2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71
support per-doc pubsub channels
2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd
Merge pull request #75 from overleaf/ho-pubsub-connection
...
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d
Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06
...
Update logger, metrics, and redis
2019-07-08 13:58:28 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Henry Oswald
eed8e215d7
bump redis-sharelatex (and io redis) to 1.0.9
2019-07-02 16:58:30 +01:00
Shane Kilkelly
e8a8c446ed
Increase timeout on test?
2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51
Update logger, metrics, and redis
...
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
2019-06-19 10:25:54 +01:00
Henry Oswald
fdef197271
Merge branch 'master' into ho-detailed-flush-status
2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c
return failed and succesfully flushed projects when flushing everything
2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c
Merge pull request #70 from overleaf/bg-metric-for-invalid-hash
...
add metric for invalid hash and other sharejs errors
2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425
Merge branch 'master' into ho-flush-all-projects
2019-05-07 17:02:14 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Tim Alby
8b40da701e
update README
...
- remove build status badge
- change app name
- update copyright notice
- update links
2019-05-07 17:03:09 +02:00
Tim Alby
d316f172bf
update repo URL for Jenkins
2019-05-07 17:03:06 +02:00
Henry Oswald
e57741cb80
stub out project flusher for unit tests
2019-05-07 15:46:30 +01:00
Henry Oswald
4b8a27a220
change github url
2019-05-07 14:01:12 +01:00
Tim Alby
ca6bfd8929
Update README
2019-05-06 17:28:52 +02:00
Henry Oswald
daca83a057
add dryRun option to flush all projects
2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e
adds /flush_all_projects project
2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9
Merge pull request #48 from sharelatex/bg-check-incoming-hash
...
check incoming hash when present
2019-04-29 10:15:44 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
33478f95fd
Fix package.json versions
2019-04-11 16:32:31 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98
check version before it is modified by applyOp
2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b
check incoming hash when present
2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Henry Oswald
4d7f9f3c1a
add sentry into settings.defaults
2019-03-05 17:34:27 +00:00
Brian Gough
11f07d1f09
Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses
...
use explicit json content-type to avoid security issues with text/html
2019-02-22 09:24:30 +00:00
Henry Oswald
937e8fe071
use redis cluster key in defaults for pendingupdates
2019-02-15 13:58:29 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
0a0fc91f28
Update app.coffee
2019-02-12 13:27:13 +00:00
Henry Oswald
3bc4cb492a
added log line
2019-02-07 16:30:53 +00:00
Henry Oswald
08723f8972
revert health check redis types
2019-02-07 15:53:26 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
b5564095f3
add /health_check/redis route back in
2019-02-07 15:10:40 +00:00
Henry Oswald
73bd264401
remove realtime keys in settings, no longer used
2019-02-07 14:55:24 +00:00
Henry Oswald
2998750a33
fix redis version lock
2019-02-06 16:01:44 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Henry Oswald
8d3fb729c5
point to web_api
2019-02-01 20:10:47 +00:00
Henry Oswald
e44a02391a
standardise name of document-updater
2019-02-01 20:04:43 +00:00
Henry Oswald
98de529e42
make config look like chef config
2019-02-01 19:52:00 +00:00
Henry Oswald
3051d7db4b
bulk upgrade to 1.1.12, moved to npm rather than git
2019-01-31 16:00:24 +00:00
Henry Oswald
d279a9149c
add dispatcherCount to config file
2019-01-25 17:24:19 +00:00
Henry Oswald
76b054c7f4
bump node redis to 1.0.5
2019-01-25 17:01:59 +00:00
Christopher Hoskin
3eb2d27d4a
Add **/*.map to .gitignore
2019-01-08 15:01:48 +00:00
Christopher Hoskin
5db1913cec
Bump settings to v1.1.0
2019-01-08 15:00:26 +00:00
Christopher Hoskin
a783c6b3cc
Bump logger to 1.5.9
2019-01-04 09:33:08 +00:00
Christopher Hoskin
05a80c7cba
Add app.map to .gitignore
2019-01-04 09:31:00 +00:00
Christopher Hoskin
0aa7315c27
Bump buildscripts to 1.1.10
2019-01-04 09:24:09 +00:00
Christopher Hoskin
d9ec90f0e2
Move to metrics v2 and shrinkwrap
2019-01-04 09:22:09 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Henry Oswald
9437cf8b02
explicity split redis config values
2018-12-05 15:16:55 +00:00
Henry Oswald
5f046ed329
install metrics route and bump metrics version
2018-12-05 12:35:46 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Henry Oswald
a11d6dcc48
Merge branch 'master' into ho-docker
2018-10-15 15:17:40 +01:00
Alasdair Smith
c128e0ab1c
Use setting instead of hard-coding port
2018-10-09 11:46:27 +01:00
Henry Oswald
ff673c71d5
set redis port via env var
2018-09-28 17:04:34 +01:00
Henry Oswald
93cf574251
upgrade build scripts 1.1.9
2018-09-28 15:13:47 +01:00
Henry Oswald
3db7377d74
add process.env["REDIS_PASSWORD"]
2018-09-28 15:11:17 +01:00
Brian Gough
537bd9a0c0
Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web
...
limit resync requests to web (connects to overleaf/sharelatex#775 )
2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d
add more logging to resync operations
2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee
limit parallel resync doc requests to web
2018-08-16 11:13:11 +01:00
Brian Gough
fdcb806518
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb
Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue"
...
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
2018-07-23 16:05:04 +01:00
Brian Gough
33e898f671
update to 1.1.9 build scripts
2018-07-23 15:35:54 +01:00
Brian Gough
6237577f87
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2
move tk call to before sandboxed module call
2018-06-05 13:54:30 +01:00
Henry Oswald
581963a4af
Dockerised env. 1.1.3 build scripts
2018-05-23 13:52:20 +01:00
Brian Gough
7e526395fb
fix check for "delete component" message
2018-04-30 09:56:01 +01:00
Brian Gough
37f431f148
also block "delete component" messages from sentry
2018-04-27 16:03:46 +01:00
Brian Gough
64670f272a
stop OpRangeNotAvailable errors going to sentry
2018-04-27 15:45:28 +01:00
Hayden Faulds
af92ca70a1
coerce projectHistoryId to integer after reading from Redis
2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49
Revert "Revert "Add projectHistoryId to updates""
2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311
Revert "Add projectHistoryId to updates"
2018-04-23 11:41:29 +01:00
Hayden Faulds
9629d0a11e
Merge pull request #26 from sharelatex/hof-deleted-project-history
...
Add projectHistoryId to updates
2018-04-23 11:01:53 +01:00
Alberto Fernández Capel
7d597fe59e
Do not specify node version in .travis.yml
...
So travis picks instead the version from .nvmrc
2018-04-19 14:03:35 +01:00
Hayden Faulds
25c0be4eaa
add projectHistryId to project structure updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32
add projectHistoryId to resync updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593
cache projectHistoryId with doc in Redis
2018-04-13 15:22:05 +01:00
Brian Gough
dd0f8b880a
change update to projectUpdate in project related methods
2018-03-16 10:54:12 +00:00
Brian Gough
779f00f912
add acceptance test for ordering of project structure changes
2018-03-15 16:27:14 +00:00
Brian Gough
3385d2640a
fix structure ordering bug
2018-03-15 16:27:10 +00:00
Brian Gough
75a5428cbf
update acceptance tests
2018-03-15 16:24:14 +00:00
Brian Gough
b3887fd984
update unit tests for incoming project versions
2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9
support project version on incoming requests
2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a
Merge pull request #21 from sharelatex/hof-history-sync
...
resync project history endpoint
2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3
extract ProjectHistoryRedisManager
2018-03-09 15:36:38 +00:00
Hayden Faulds
fc6ef6ea7a
improve logging
2018-03-09 12:21:48 +00:00
Hayden Faulds
088ab6e7ba
rename queueResyncDocContents -> queueResyncDocContent
2018-03-09 12:21:37 +00:00
Hayden Faulds
64d1fe13bc
additional logging for project history resync
2018-03-07 16:24:22 +00:00
Hayden Faulds
a72d228354
add unit tests
2018-03-07 13:29:53 +00:00
Hayden Faulds
152a25e96d
rename resyncProject -> resyncProjectHistory
2018-03-07 13:29:42 +00:00
Hayden Faulds
039f997e80
update project history resync url
2018-03-07 11:37:26 +00:00
Hayden Faulds
d46217be52
queue doc content sync updates
2018-03-02 15:02:27 +00:00
Hayden Faulds
52d6b710bd
add ability to resync project structure
2018-03-02 15:01:53 +00:00
James Allen
939eaa2d4b
Don't allow a document to be loaded without a pathname
2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3
remove unecessary method wrapping
2018-02-22 10:17:59 +00:00
Hayden Faulds
0f87ae1f74
simplify comments
2018-02-22 10:16:29 +00:00
Hayden Faulds
ea0dd9700b
reduce the number of times we flush project history
2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d
Merge pull request #18 from sharelatex/msw-flush-project-ops
...
Flush project structure changes every 500
2018-02-16 10:30:00 +00:00
James Allen
5425fdef8a
update to 1.0.1 build scripts
2018-02-15 16:54:21 +00:00
James Allen
657da70d45
Fix unit test namespacing from mocha upgrade
2018-02-15 16:28:58 +00:00
James Allen
5d1659457e
Update to build 1.0.0 build scripts
2018-02-15 16:28:40 +00:00
James Allen
f6c79f3203
Fix and extend unit tests
2018-02-15 15:55:12 +00:00
Michael Walker
6c57317f8d
Add a test that sending too few updates does not flush history
2018-01-31 11:41:08 +00:00
Michael Walker
241d1b27d5
Remove _shouldFlushHistoryOps wrapper
2018-01-31 11:27:40 +00:00
Michael Walker
772ee4083a
Turn a unit test into an acceptance test
2018-01-31 11:17:56 +00:00
Michael Walker
bdaa2ffa77
Add an acceptance test for flushing project changes
2018-01-26 14:41:47 +00:00
Michael Walker
d2865f0650
Correctly count project_ops_length
2018-01-26 14:41:28 +00:00
Michael Walker
3028fb9c3d
Only flush project updates when crossing the threshold
2018-01-24 11:57:03 +00:00
Michael Walker
24c74db0dc
Flush project changes when queue hits limit
2018-01-23 11:18:20 +00:00
Michael Walker
3d313ebc53
Add method to get number of queued project updates
2018-01-23 11:10:54 +00:00
James Allen
ccf6cb1a18
Provide hosts as environment settings, add npm run start script and update coffeescript
2017-12-29 08:13:16 +00:00
Brian Gough
0b583dd4e6
fix log message
2017-12-20 13:27:10 +00:00
Brian Gough
a8b14552c0
allow flush to succeed when doc is already deleted
2017-12-19 12:27:57 +00:00
Brian Gough
41f15c4fa8
fix unit test
2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf
Merge branch 'bg-limit-ops-in-lock'
2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d
Calculate and send doc length before each update
2017-12-11 17:26:32 +00:00
Hayden Faulds
bd45fcb5f8
Merge pull request #12 from sharelatex/hof-version-entity-creation
...
version entity creation; Connects to sharelatex/web-sharelatex-internal#104
2017-11-24 10:02:51 +00:00
Brian Gough
fb572670cd
Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects
...
use a separate parameter for doc and project flush
2017-11-21 13:54:03 +00:00
Brian Gough
0bb9f147aa
fix whitespace
2017-11-13 11:56:08 +00:00
Brian Gough
a4bbf5cf52
use a separate parameter for doc and project flush
2017-11-13 11:53:39 +00:00
Hayden Faulds
d0e56e5054
add acceptance tests for entity additions
2017-11-10 15:01:37 +00:00
Hayden Faulds
a3420b1236
version entity additions
2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac
add acceptance test for entity renaming
2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e
version file renames
2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23
version document renames
2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612
allow settings.redis.project_history to be undefined
2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79
Merge pull request #2 from sharelatex/hof-doc-pathname
...
Add metadata to updates for history
2017-10-25 13:13:00 +01:00
Brian Gough
e590543e42
Merge pull request #4 from sharelatex/bg-fix-listen-in-acceptance-tests
...
exit if mock servers fail to start
2017-10-25 09:13:48 +01:00
Brian Gough
f3098f7470
upgrade ioredis to 3.2.1 via redis-sharelatex
2017-10-24 12:20:14 +01:00
Brian Gough
367ed1e941
Merge pull request #8 from sharelatex/bg-upgrade-to-node-6x
...
upgrade to node 6.9.5
2017-10-24 11:11:23 +01:00
Brian Gough
56fd6e3d48
upgrade to node 6.9.5
2017-10-24 11:08:20 +01:00
Brian Gough
d9d58393d7
remove unused ioredis package
...
it is loaded from redis-sharelatex, not here.
2017-10-24 10:39:00 +01:00
Hayden Faulds
d1f0c4ceae
add acceptance tests for project history API
2017-10-24 09:06:23 +01:00
Brian Gough
14f3d22071
upgrade to latest async ^2.5.0
2017-10-23 16:02:24 +01:00
Brian Gough
20655b6eeb
remove unused requires of async
2017-10-23 15:59:32 +01:00
Hayden Faulds
04ecd1e7ee
set pathname in acceptance tests
2017-10-23 15:32:25 +01:00
Hayden Faulds
be41a1614d
decorate ops with pathname and doc_length
2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8
flush track-changes and project-history in HistoryManager
2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c
conditionally enqueue history updates for project
2017-10-23 15:32:25 +01:00
Hayden Faulds
c27df0bfef
split apart multi and rclient in tests
2017-10-23 14:24:23 +01:00
Hayden Faulds
81f998afe9
return pathname from DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc
handle pathname in DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386
store pathname in Redis cache
2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a
return pathname from PersistenceManager
2017-10-23 14:17:14 +01:00
Brian Gough
02d3d1bd17
fix unit tests
...
timekeeper must be called before SandboxedModule.require
2017-10-23 14:02:53 +01:00
Brian Gough
7c2a28c161
fix use of timekeeper in unit tests
2017-10-23 13:33:14 +01:00
Brian Gough
4d82f1432a
Merge pull request #5 from sharelatex/bg-upgrade-ioredis-to-3x
...
update to ioredis 3
2017-10-23 13:27:02 +01:00
Brian Gough
448f131011
exit if mock servers fail to start
2017-10-20 14:56:12 +01:00
Joe Green
3df2d60948
Update Jenkinsfile
2017-10-16 14:14:06 +01:00
Joe Green
d387f979bc
only alert on master
2017-10-12 16:53:31 +01:00
Brian Gough
3954ecf85e
add missing timekeeper package
2017-10-12 11:23:24 +01:00
Brian Gough
08f0c67cbf
Merge pull request #3 from sharelatex/bg-auto-doc-flush
...
add unflushed time to doc in redis; connects to overleaf/sharelatex#190
2017-10-12 10:55:54 +01:00
Brian Gough
79c276ea77
update to ioredis 3
2017-10-11 17:00:01 +01:00
Brian Gough
727a534d01
keep the existing GET method temporarily
...
to avoid any problems due to deployment in the wrong order
2017-10-11 15:44:35 +01:00
Brian Gough
7cbb3e7af8
change getProjectDocs endpoint from GET to POST
...
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c
update DocumentManager.getDoc signature
...
keep alreadyLoaded flag at the end for consistency
2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005
add unflushed time to doc in redis
2017-10-06 12:34:28 +01:00
James Allen
340aa98de5
Add OSS syncing task
2017-10-04 11:27:19 +01:00
Brian Gough
ad745f133d
Merge pull request #59 from sharelatex/bg-reset-project-state
...
add endpoint to clear project state
2017-09-13 10:39:47 +01:00
Brian Gough
3842a27278
change .../clear endpoint to .../clearState
2017-09-12 11:39:18 +01:00
Brian Gough
2047bd5516
fix broken unit test
2017-09-11 09:44:05 +01:00
Brian Gough
f34c12fdc7
add endpoint to clear project state
2017-09-08 15:50:26 +01:00
Brian Gough
215a939d4e
add comment about log.error for excludeVersions
2017-09-08 13:43:22 +01:00
Brian Gough
1e54b4c72c
log all errors in getProjectDocs
2017-09-07 14:12:48 +01:00
Joe Green
c16c6c3bd3
added build.txt
2017-09-04 14:48:47 +01:00
Brian Gough
9d931a3fa7
fix broken unit tests
2017-08-18 12:08:39 +01:00
Brian Gough
230c93766d
fix broken RedisManager unit test
2017-08-18 11:59:31 +01:00
Brian Gough
8d408de875
Merge pull request #56 from sharelatex/bg-compile-from-redis
...
provide endpoint for current project docs in redis
2017-08-18 11:33:27 +01:00
Joe Green
0e8ce29414
compile acceptance tests
2017-08-14 15:33:30 +01:00
Joe Green
b1a7f779d1
add acceptance tests
2017-08-14 15:28:04 +01:00
Brian Gough
8372911a1b
return doclines as array from getProjectDocs
...
for compatibility with getDoc
2017-08-11 16:56:30 +01:00
Brian Gough
5971a19084
added acceptance tests
2017-08-11 16:55:31 +01:00
Brian Gough
f6be68eb08
added missing unit tests file
2017-08-11 16:49:16 +01:00
Joe Green
1146253c0a
Create Jenkinsfile
2017-08-11 11:06:36 +01:00
Brian Gough
f696ccb0d9
add unit tests for getProjectDocs
2017-08-10 16:41:28 +01:00
Brian Gough
1eb80936f2
fix log line
2017-08-10 14:57:27 +01:00
Brian Gough
47bebf9638
no need to support optional keys.projectState
2017-08-09 16:47:09 +01:00
Brian Gough
bd6b5b2c59
use v instead of rev in getProjectDocs
2017-08-09 15:30:53 +01:00
Brian Gough
ca7d9dce85
fix unit tests
2017-08-08 09:40:39 +01:00
Brian Gough
9f3ec72f81
switch to single get/set method for getProjectDocs
...
if project state hasn't changed, return the docs. Otherwise set the hash
and return a 409 Conflict response.
2017-08-07 14:43:28 +01:00
Brian Gough
f5f516a910
delete clsi state when deleting doc (if used)
2017-08-03 15:04:12 +01:00
Brian Gough
860537146b
avoid logging doclines in getProjectDocs
2017-08-03 15:04:12 +01:00
Brian Gough
80be5adc0d
change filter to exclude in getProjectDocs
2017-08-03 15:04:06 +01:00
Brian Gough
54e02cd895
update docupdater endpoint to /project/id/docs
2017-08-03 14:41:34 +01:00
Brian Gough
0109585092
provide endpoint for current project docs in redis
2017-08-02 13:42:10 +01:00
Shane Kilkelly
c3465cf3d4
Reduce retry delay to 10ms, and refactor
2017-07-18 15:28:18 +01:00
Shane Kilkelly
dc8d08c030
Update acceptance tests to reflect retry logic
2017-07-17 11:13:21 +01:00
Shane Kilkelly
0d46c3e2b3
WIP: auto-retry web requests
2017-07-17 10:08:21 +01:00
Brian Gough
c21cc013f3
use null instead of err in callbacks on success
2017-07-12 10:48:32 +01:00
Brian Gough
d2730c8d6b
unit tests for locking timeouts
2017-07-12 10:48:32 +01:00
Brian Gough
bb0dc4b4d5
release the lock if it took too long to acquire it
2017-07-12 10:48:32 +01:00
Brian Gough
54c0fc1180
put a 5 second timeout in getting redis lock
2017-07-11 14:28:58 +01:00
Brian Gough
ffea31290c
return total time from the profile .end() method
2017-07-11 14:28:58 +01:00
Brian Gough
41c2bd7fcd
Merge pull request #53 from sharelatex/bg-timeout-on-slow-prev-ops
...
add a timeout for getPreviousDocOps
2017-06-27 13:26:02 +01:00
Brian Gough
59ace9d5c5
avoid long lines in timeout check
2017-06-26 16:36:05 +01:00
Brian Gough
8e1e14c9ca
update to redis-sharelatex 1.0.3
...
adds keepalive support
2017-06-26 13:49:44 +01:00
Brian Gough
963e513057
add a timeout for getPreviousDocOps
...
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
2017-06-23 15:50:21 +01:00
James Allen
7b6966899b
Add script to expire existing DocOps lists
2017-06-14 12:09:34 +01:00
James Allen
54f128289f
Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors
...
add metric for unlock errors
2017-06-13 14:46:40 +01:00
James Allen
7d22e4422c
Merge pull request #49 from sharelatex/bg-add-redis-timeout
...
add redis timeout
2017-06-13 14:46:32 +01:00
James Allen
30f57a7bfc
Merge pull request #51 from sharelatex/ja-expires-ordering
...
Ensure expires is called after the key is created
2017-06-13 14:05:22 +01:00
James Allen
770b53f6af
Add missing mocks to speed up tests and clean up output
2017-06-08 16:43:27 +01:00
James Allen
a3d7260619
Ensure expires is called after the key is created
2017-06-08 16:43:06 +01:00
Brian Gough
2e5d57cd99
don't increase rate limit when tasks are failing
2017-06-07 11:47:28 +01:00
Brian Gough
1ef258c878
add a timeout on the redis getDoc request
2017-06-07 11:34:42 +01:00
Brian Gough
673f4228ce
add metric for unlock errors
...
sentry does not record them reliably, due to rate limiting
2017-06-05 16:29:58 +01:00
Brian Gough
729216c9b1
add unit tests for DocsIn check
2017-05-31 16:08:45 +01:00
Brian Gough
12e8eaa9b6
fix bug in doclines check
2017-05-31 16:08:33 +01:00
Brian Gough
bed64d26c8
check if doc is missing from DocsIn set
2017-05-31 15:33:59 +01:00
Brian Gough
fa089dc28d
avoid graphite drift in active worker count
2017-05-30 14:37:01 +01:00
Brian Gough
891ffda3bf
put a rate limit on worker calls
2017-05-26 15:08:35 +01:00
Brian Gough
c43f97e8e3
add metric for active workers
2017-05-24 12:12:15 +01:00
Brian Gough
a3947c587f
fix bug, jsonOps is array not string
2017-05-22 15:20:28 +01:00
Brian Gough
54ad45c870
add null byte check for ranges
2017-05-22 15:00:05 +01:00
Brian Gough
a13f055d62
add null byte check for ops
2017-05-22 15:00:00 +01:00
Brian Gough
78828ebbc5
fine grained logging for cpu usage in applyUpdate
2017-05-19 16:00:16 +01:00
Brian Gough
a550078071
replace verbose logging by slow query log
2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4
clarify calculation of process.hrtime
2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e
log profilers start and end times for reference
2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a
increase profiling cutoff to 1s
2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1
add profiling
2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb
log for each get/try/release lock
2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da
bug fix in backoff
2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a
exponential backoff for lock up to maximum time
2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796
Revert "fail safely if doc cannot be loaded"
2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4
Merge pull request #34 from sharelatex/pr-bulk-actions
...
Add methods to bulk accept changes.
2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257
Update ranges tracker.
2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6
Merge branch 'master' into pr-bulk-actions
2017-05-15 10:35:16 +01:00
Paulo Reis
36d16a76e0
Unit test accept changes in the ranges manager.
2017-05-12 14:42:40 +01:00
Brian Gough
2ee40d0748
ensure document is in redis before consuming ops
2017-05-12 14:22:18 +01:00
Brian Gough
be96548199
log number of updates for future debugging
2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750
put a limit on the number of ops per iteration
2017-05-12 13:15:59 +01:00
Paulo Reis
d53266ea2d
Update web API stub.
2017-05-11 12:00:34 +01:00
Paulo Reis
5677c7ad9a
Update range tracker.
2017-05-11 11:31:19 +01:00
Paulo Reis
f690ef078a
Update tests; add new ones for multiple changes.
2017-05-09 16:16:25 +01:00
Brian Gough
36407ac726
rename HistoryManager pushUncompressedHistoryOps
2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d
increase flush threshold to 100 ops
2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8
get history ops length directly from redis update
2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7
rename pushUncompressedHistoryOps
2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516
Update tests with new methods.
2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054
Use new bulk method.
2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80
Add bulk method from range tracker.
2017-05-08 16:08:34 +01:00
Brian Gough
e2f70aca1a
fix tests for redis cluster
2017-05-08 16:02:49 +01:00
Brian Gough
79d8fced49
make history update more atomic
2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29
Remove unused method.
2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030
Consolidate methods to accept changes.
2017-05-05 15:12:06 +01:00
James Allen
87a5e30a0b
Bump redis-sharelatex version
2017-05-05 13:37:58 +01:00
James Allen
7456238a71
Bump redis-sharelatex version
2017-05-04 15:42:10 +01:00
Paulo Reis
d530ee9501
Add methods to bulk accept changes.
2017-05-04 15:32:54 +01:00
James Allen
5f93640077
Add scripts for testing cluster failover scenarios
2017-05-04 11:14:17 +01:00
James Allen
4104ca4889
Add in separate redis config for the lock and fix a few web -> realtime
2017-05-03 16:27:32 +01:00
James Allen
5c3661d6d9
Merge remote-tracking branch 'origin/bg-redis-cluster-refactor' into ja-redis-cluster-refactor
2017-05-02 15:39:54 +01:00
James Allen
ed493d8ad3
Rename web -> realtime for consistency with realtime
2017-05-02 15:38:33 +01:00
Brian Gough
64aef0b55a
fix acceptance test to work with redis cluster too
2017-04-27 10:42:43 +01:00
Brian Gough
dc77bc207d
change acceptance test to use redis history client
2017-04-24 16:31:23 +01:00
James Allen
c5449ae282
Split out redis config for real-time and track-changes into separate cluster-compatible configs
2017-04-13 17:00:42 +01:00
James Allen
f21208e841
Use new redis-sharelatex instead of RedisBackend for cluster abstraction
2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05
fix callback prototype in getLock
2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475
improve unlock error handling
2017-03-30 15:31:34 +01:00
Brian Gough
b2b4bc44df
upgrade logger-sharelatex to v1.5.6
2017-03-30 14:16:35 +01:00
Brian Gough
527c02a00b
log op versions pushed to track changes
2017-03-30 11:20:50 +01:00
Shane Kilkelly
64658d0034
Add a .nvmrc file
2017-03-27 14:50:09 +01:00
James Allen
0245bfd031
Support an incoming undo flag for history restores
2017-03-21 11:20:38 +00:00
James Allen
a3a5524778
Fix broken unit test from merge
2017-03-20 10:37:02 +00:00
James Allen
199079a5b7
Merge branch 'master' into ja-cut-and-paste-comments
2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c
Update RangeTracker to support upserting and moving comments for cut and paste
2017-03-16 15:49:41 +00:00
James Allen
5499a67d78
Add in a consistency check after applying updates that ranges still match
2017-03-15 14:12:06 +00:00
James Allen
b9e4492001
Wrap RangeTracker in try catch
2017-03-06 19:05:11 +00:00
James Allen
a3a9a726d8
Update unit tests
2017-03-06 12:01:18 +00:00
James Allen
d8135749d2
Merge branch 'master' into ja-increase-limits-on-changes
...
Conflicts:
app/coffee/RedisManager.coffee
2017-03-06 11:57:40 +00:00
James Allen
30294dafb1
Merge pull request #23 from sharelatex/bg-upgrade-raven-logging
...
upgrade to logger-sharelatex 1.5.3
2017-03-06 11:53:47 +00:00
James Allen
269433f250
Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch
...
log doclines on hash mismatch
2017-03-06 11:53:24 +00:00
Brian Gough
501d907299
upgrade to logger-sharelatex 1.5.4
2017-03-03 16:08:14 +00:00
Brian Gough
d086e0b61b
log doclines on hash mismatch
2017-03-03 15:57:44 +00:00
Brian Gough
3f13263ecf
upgrade to logger-sharelatex 1.5.3
2017-03-03 15:32:11 +00:00
Brian Gough
b186a01c04
don't log errors from redis backend
...
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
2017-03-03 15:27:42 +00:00
James Allen
d56bb55953
Revert PR #19
2017-03-01 16:49:46 +00:00
James Allen
9cfe651930
Merge pull request #19 from sharelatex/ja-track-changes
...
Don't consume partial parts of delete when inserting a change
2017-03-01 10:20:06 +01:00
James Allen
f544814dda
Add in an absolute size limit on the ranges JSON object
2017-02-27 14:34:20 +01:00
James Allen
f211c282b2
Merge branch 'master' into ja-increase-limits-on-changes
2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d
Improve logging around hashes and null byte errors
2017-02-27 10:06:24 +01:00
James Allen
60f3393c44
Do a naive check for null bytes after JSON.stringify on doc lines
2017-02-27 09:55:04 +01:00
James Allen
fd3dee0af5
Increase changes count limit to 2000
2017-02-24 14:58:47 +01:00
James Allen
80284e1b01
Only cancel deletes with inserts on undo and reject
2017-02-24 14:21:06 +01:00
James Allen
0420929e2a
Reduce the data logged when updating ranges
2017-02-23 13:57:47 +01:00
James Allen
e1481df76a
Don't consume partial parts of delete when inserting a change
2017-02-23 11:39:19 +01:00
Brian Gough
49271d4ee4
update logger to 1.5.2 - rate limiting on sentry
2017-02-23 10:15:56 +00:00
James Allen
5ab903169a
evert "update logger to 1.5.2 - rate limiting on sentry"
...
This reverts commit ba1c4a84959cd2eb33d7d624239d3e621ebae1fd.
2017-02-22 18:26:42 +01:00
James Allen
d6b5cc6389
Increase changes limit to 1200
2017-02-22 18:11:55 +01:00
Brian Gough
ed59ca32fd
update logger to 1.5.2 - rate limiting on sentry
2017-02-22 15:58:40 +00:00
Brian Gough
168bab6960
make hash error logging fine grained
2017-02-22 14:32:51 +00:00
Brian Gough
19e7522d3f
make logging of hash errors optional
2017-02-22 13:46:34 +00:00
Brian Gough
8d1b11f37a
Merge branch 'bg-verify-writes-II'
2017-02-21 16:03:48 +00:00
Brian Gough
5eb339e525
fix utf8 encoding in sha1 hash
2017-02-21 16:03:06 +00:00
Brian Gough
6b808522dc
Merge pull request #18 from sharelatex/bg-verify-writes
...
merge redis hash check
2017-02-20 15:55:19 +00:00
Brian Gough
590f8e7ced
Merge pull request #15 from sharelatex/bg-verify-writes
...
store sha1 hash of docLines in redis
2017-02-20 15:53:56 +00:00
Brian Gough
62165ddeab
add unit tests
2017-02-20 15:33:19 +00:00
Brian Gough
c57d3ce31c
compute hash on write in redis server
2017-02-20 14:09:09 +00:00
Brian Gough
a3c127e469
added unit tests
2017-02-20 13:53:25 +00:00
Shane Kilkelly
37172a2115
Debug Acceptance Tests: Try without npm rebuild
2017-02-20 09:22:54 +00:00
Shane Kilkelly
28536bfe73
try running mochatest directly without compiling
2017-02-17 16:14:13 +00:00
James Allen
660752dfa6
Increase change limit to 1000
2017-02-17 15:32:44 +01:00
James Allen
89537927b5
Update Ranges Tracker
2017-02-17 09:24:19 +01:00
James Allen
de7f760ea4
Update Ranges Tracker
2017-02-16 17:09:08 +01:00
Brian Gough
b5a4458b68
check sha1 hash value only if present
2017-02-15 14:12:36 +00:00
Brian Gough
bd70aaa76c
add sha1 hash support on writes
2017-02-14 16:11:43 +00:00
Brian Gough
a33d4f505b
log request object on error
2017-02-14 15:20:05 +00:00
Brian Gough
8d1aa56689
enable sentry for docupdater
2017-02-14 14:34:28 +00:00
James Allen
8725aeab03
Allow deleting of comment ranges
2017-01-24 15:57:11 +01:00
James Allen
dc976fd9e9
Use thread id for comment id
2017-01-23 09:45:46 +01:00
James Allen
0705242a55
Increase max op age to 80
2017-01-17 11:45:10 +01:00
James Allen
5fed2424d0
Remove unused redis package reference
2017-01-16 13:05:05 +01:00
James Allen
0706feb26b
Add max limit on number of comments and changes per doc
2017-01-10 16:58:11 +01:00
James Allen
a3d5971d54
Update RangesTRacker
2017-01-10 11:59:09 +01:00
James Allen
540d0f7ec7
Make sure comment ids are unique
2017-01-10 11:55:38 +01:00
James Allen
be19532a1d
Add HTTP end point for accepting changes
2017-01-09 14:41:18 +01:00
James Allen
65f4360738
Consolidate HttpController tests into one file
2017-01-09 14:34:10 +01:00
James Allen
593e7260d4
Update RangesTracker
2017-01-09 10:52:06 +01:00
James Allen
7cac2f7d76
Generate deterministic range ids based on seed
2017-01-09 10:46:58 +01:00
James Allen
2c7029cc50
Revert "Support a {dr:...} op for deleting ranges"
...
This reverts commit 24c58e5ad430e0240533cc1e5c21122859fe8dc9.
2017-01-09 09:24:19 +01:00
James Allen
0f13cb3aa7
Support a {dr:...} op for deleting ranges
2017-01-06 16:58:51 +01:00
James Allen
59a06cd798
Accept comments with thread id as an op type
2016-12-13 15:51:47 +00:00
James Allen
47b19818ff
Add in new comment op type
2016-12-12 17:53:43 +00:00
James Allen
e3fee1a1d1
Rename 'track changes entries' -> 'ranges'
2016-12-08 12:31:43 +00:00
James Allen
3ea2e07993
Add tests for fetching and flushing track changes entries to persistence layer
2016-12-02 12:01:23 +00:00
James Allen
418405e8b9
Return track changes entries in HTTP request
2016-12-02 11:37:27 +00:00
James Allen
4fadd75ef3
Track changes based on flag on op, not global setting
2016-12-02 11:04:21 +00:00
James Allen
f43355b74d
Fix UpdateManager tests
2016-12-01 18:19:47 +00:00
James Allen
ce93a76e7f
Fix ShareJsUpdateManager tests
2016-12-01 18:11:03 +00:00
James Allen
e7ff05e792
Fix UpdateManager tests
2016-12-01 18:06:33 +00:00
James Allen
889f5fdf9f
Fix ShareJsDB tests
2016-12-01 17:14:40 +00:00
James Allen
b6c93c718d
Update TrackChangesManagerTests -> HistoryManagerTests
2016-12-01 16:50:55 +00:00
James Allen
d878dd5758
Fix RedisManagerTests
2016-12-01 16:49:53 +00:00
James Allen
9ee913be39
Update PersistenceManagerTests
2016-12-01 16:40:15 +00:00
James Allen
fb39e37fe0
Update DocumentManager tests
2016-12-01 16:27:40 +00:00
James Allen
d36f1a86f1
Merge branch 'master' into ja-track-changes
...
Conflicts:
app/coffee/PersistenceManager.coffee
2016-12-01 16:06:13 +00:00
James Allen
efab68e6b2
Add in guards on bad data from web API
2016-11-29 17:13:16 +00:00
James Allen
546d9e9cee
Remove left over test
2016-11-29 17:13:07 +00:00
James Allen
2852043a6d
Don't store doc version in Mongo directly, instead use docstore
2016-11-29 17:06:23 +00:00
James Allen
77b1d0ea21
Fix up package versions
2016-11-29 15:29:22 +00:00
James Allen
f7ba0946be
More acceptance test fixes
2016-11-29 15:09:11 +00:00
James Allen
2df5c08389
Update acceptance tests to use redis-sharelatex
2016-11-29 15:05:02 +00:00
James Allen
db8b4bf991
Update acceptance test script
2016-11-29 14:57:05 +00:00
James Allen
e739e86c48
Get basic ChangeTracker hooked up. WIP
2016-11-28 10:14:42 +00:00
James Allen
5ce15c4d60
Move check of zero length op array up a level
2016-09-12 10:37:41 +01:00
James Allen
993aab7a78
Don't try to redis rpush with no arguments
2016-09-12 10:37:32 +01:00
James Allen
6ca48523d9
mapSeries -> eachSeries
2016-09-09 15:28:27 +01:00
James Allen
2d82d56f93
Process updates one at a time and write into redis after each one
2016-09-09 11:01:14 +01:00
James Allen
c85fd7ef41
Merge branch 'master' into ja-consistency-refactor
2016-09-07 11:08:04 +01:00
Brian Gough
8d14f8f7f0
add route parameters
2016-09-05 10:32:11 +01:00
Brian Gough
9bc7594226
clean up redis query
2016-09-05 10:31:54 +01:00
James Allen
8779f3f686
Only write DocOps atomically with version and lines, after all docs are applied
2016-08-23 16:00:46 +01:00
James Allen
87f3e5e809
Remove old migration code
2016-08-23 11:03:37 +01:00
James Allen
cb62b005f3
Add in event loop monitoring
2016-08-23 09:53:17 +01:00
Shane Kilkelly
39cc9cda6d
Remove docker-related config. test:acceptance:docker
now depends on acceptance-test-runner-sharelatex
.
2016-07-18 11:09:04 +01:00
Shane Kilkelly
02b7b7baf2
Merge branch 'master' of github.com:sharelatex/document-updater-sharelatex
2016-07-18 10:36:12 +01:00
Shane Kilkelly
c446b20375
Add alternative test:acceptance:docker command, using separate docker repo
2016-07-18 10:36:03 +01:00
James Allen
a1bac9719a
Update secondary timeout for multi commands too
2016-07-12 11:45:10 +01:00
James Allen
d0c54f1be6
Increase secondary timeout to 600ms
2016-07-12 11:11:55 +01:00
Shane Kilkelly
4c0dc5a0ef
Increase timeout for test.
2016-07-08 16:05:36 +01:00
Shane Kilkelly
0a5f95f22d
Move docker-entrypoint.sh into the test/acceptance/scripts directory
2016-07-08 15:52:02 +01:00
Shane Kilkelly
32d06b805c
run only the acceptance tests inside container, rather than compile and test.
2016-07-08 15:15:41 +01:00
Shane Kilkelly
b16b34d7c6
only execute app inside container, instead of compile and execute.
2016-07-08 14:50:30 +01:00
Shane Kilkelly
6d9aecae4c
capture exit code of the test run.
2016-07-08 14:31:43 +01:00
Shane Kilkelly
ea41453442
fix docker build command.
2016-07-08 13:44:43 +01:00
Shane Kilkelly
28fda02d2d
Merge branch 'master' into sk-dockerize-tests
2016-07-08 13:37:04 +01:00
Shane Kilkelly
879482a955
grunt scripts to run the docker acceptance tests
2016-07-08 12:09:01 +01:00
James Allen
4f34cb7363
Only set keys in migration if they exist
2016-07-08 11:45:36 +01:00
James Allen
bd392f5682
Comment out cluster config (again...)
2016-07-08 10:54:56 +01:00
Shane Kilkelly
bca8f6c376
working docker image which can run the acceptance tests
2016-07-08 10:51:22 +01:00
James Allen
8b090c0a60
Don't call sadd with blank args
2016-07-07 16:41:38 +01:00
James Allen
9bd7c0017d
Allow migrate script to process multi docs at once
2016-07-07 16:29:01 +01:00
Shane Kilkelly
c38d903f92
Basic docker file.
2016-07-07 16:07:14 +01:00
James Allen
6bc78ccf77
Add in some null checks on comparing backend results
2016-07-07 11:24:33 +01:00
James Allen
1c62a1c5a1
Don't propagate error on secondary timeout
2016-07-07 09:46:15 +01:00
James Allen
edf1e1ab49
Initialize primaryResult to an array, not null
2016-07-06 15:02:05 +01:00
James Allen
59883023ca
Ignore different order of results from smembers
2016-07-06 14:24:27 +01:00
James Allen
89f90c1b04
Timeout secondary requests if they take longer than 200ms
2016-07-06 12:25:36 +01:00
James Allen
508a95c19b
Use the main redis instance for locks
2016-07-06 11:50:02 +01:00
James Allen
e912ccc562
Wait for both backends to return so that they are always in sync
2016-07-05 16:07:47 +01:00
James Allen
dfd45bd23c
Add timers to time how long each redis request takes
2016-07-04 11:14:23 +01:00
James Allen
149351aa49
Remove debugging log lines
2016-06-29 21:18:13 +01:00
James Allen
fde334acc4
Create script that will ensure data is migrate between redis backends
2016-06-29 12:57:56 +01:00
James Allen
dbe03e2757
Track metrics of when backends match or disagree
2016-06-28 16:49:15 +01:00
James Allen
0b9e85ea50
Update ioredis version
2016-06-28 15:47:06 +01:00
James Allen
e04c946ecd
Fix unit tests
2016-06-23 18:04:26 +01:00
James Allen
da89ff7172
Add in external health check rather than internal
2016-06-23 18:00:03 +01:00
James Allen
414ab5d6a9
Refactor TCP redis monitoring to be more explicit
2016-06-23 15:59:25 +01:00
James Allen
8ef03c3d2f
Add in application layer monitoring of the health of each cluster node
2016-06-23 15:38:51 +01:00
James Allen
27a74d6b71
Improve consistency of stress test script
2016-06-21 14:31:20 +01:00
James Allen
392beac57b
Improve error reporting and retry ops in stress test script
2016-06-20 15:01:39 +01:00
James Allen
7915c2ab6c
Tweak stress test to work on server
2016-06-20 13:56:46 +01:00
James Allen
39d920682a
Only pass through keys to redis that are explicitly set in the config
2016-06-17 14:11:22 +01:00
James Allen
40b7da8edb
Support endpoints and masterName in redis config
2016-06-17 14:05:54 +01:00
James Allen
e5cf856ddf
Add in stress test script
2016-06-17 12:17:43 +01:00
James Allen
bc00aab7b1
Explicitly separate calls to web and docupdater redis instances
2016-06-17 12:17:22 +01:00
James Allen
9ff3026807
Merge branch 'master' into ja_redis_cluster
2016-06-09 10:00:58 +01:00
James Allen
48a92b28e5
Don't run redis commands in parallel for easier consistency reasoning
2016-06-09 09:54:13 +01:00
James Allen
c823e06912
Don't run redis commands in parallel for easier consistency reasoning
2016-06-08 16:42:09 +01:00
James Allen
c6605ed5f0
Fix misapplied arguments to multi commands
2016-06-08 16:41:58 +01:00
James Allen
437e885812
Lock down to specific async version
2016-06-08 16:21:56 +01:00
James Allen
b4936f62af
Check that return values from different redis backends match
2016-06-08 12:18:37 +01:00
James Allen
ef43e2b325
Configure backend database as redis cluster
2016-06-07 18:38:32 +01:00
James Allen
1db6f8f159
Pull out rclient into RedisBackend that supports sending requests to multiple rclients
2016-06-07 17:58:18 +01:00
James Allen
e38e883081
Clean up and consolidate RedisManager tests for easy refactoring later
2016-06-01 12:22:47 +01:00
James Allen
ea8f4a6d7a
Remove unused getDocOpsLength method
2016-06-01 11:58:15 +01:00
James Allen
8004e022fe
Fix async version and fix affected unit tests
2016-06-01 11:49:24 +01:00
James Allen
05b09a447e
Ensure that all multi call keys will hash to the same node in cluster
2016-06-01 11:40:12 +01:00
James Allen
f707783aba
Remove old methods of triggering doc updates
...
Remove the old pub/sub listener which is no longer used.
Also remove the DocsWithPendingUpdates set, which used to track
docs waiting to be updated. This was necessary incase messages were
missed on the pub/sub channel, so we knew which docs still had pending
updates. However, now we use the BLPOP queue, these updates just sit in
the queue until a consumer comes back to continue consuming them.
2016-06-01 11:28:23 +01:00
James Allen
b2e58266ae
Remove unused total doc counting redis commands
2016-06-01 11:03:39 +01:00
James Allen
32ce44f9c6
Remove unused gzipping logic
2016-06-01 10:53:50 +01:00
James Allen
1d7cbd8a78
Fix unit tests
2016-05-31 13:32:15 +01:00
James Allen
11667e8250
Return semantic error if the requested ops are not available when getting a document
2016-05-31 13:24:19 +01:00
Shane Kilkelly
048fd19418
Add one to size of line.
...
To account for newline characters in the original document
2016-05-12 09:26:50 +01:00
Shane Kilkelly
82d5a7fafd
check total size of lines, rather than content-length
2016-05-11 15:55:21 +01:00
Shane Kilkelly
210a61112b
reject payloads larger than 2MB for setDoc
2016-04-29 15:08:21 +01:00
James Allen
4ab4e27724
Include COUNT in lock and make RND per-process incase randomBytes blocks
2016-04-13 16:27:53 +01:00
James Allen
d15a22be26
Remove misleading log line about lock value
2016-04-13 12:20:53 +01:00
James Allen
945c728db2
Use signed locks so only the locking party can remove their lock
2016-04-13 11:59:56 +01:00
James Allen
6c79ab4321
Don't let HTTP calls take longer than 5 seconds since we're inside a 30 second lock
2016-04-12 17:10:39 +01:00
James Allen
3c7c318ea0
Clean up and speed up unit tests by making sure requires are mocked
2016-01-20 17:36:06 +00:00
James Allen
af5d01e440
Flush track changes when unloading data from redis
2016-01-20 15:05:31 +00:00
James Allen
184c9031cd
Delete doc from redis after a full set if it wasn't in redis before
2016-01-20 14:31:25 +00:00
Brian Gough
e8f09f3357
added comment about flushing to track changes
2015-12-01 16:03:05 +00:00
Brian Gough
6e97521971
Revert "Merge pull request #4 from sharelatex/flush-ops-from-redis"
...
This reverts commit 84a9ad8b67b860a92271cf7bb669cf21cc0b5b86, reversing
changes made to a6d4649f4f2db06862e8cbc5e132a073ce8c20b2.
2015-12-01 15:58:40 +00:00
Brian Gough
13e22e1802
Revert "only flush to track-changes when ops are queued"
...
This reverts commit 0baa8f989481ca263fa71e90af21f13b29c29504.
2015-12-01 15:48:49 +00:00
Brian Gough
64f5d5526e
Revert "update tests"
...
This reverts commit 501e891760fd5c5bd0e6ffc75f866fd565561f31.
2015-12-01 15:48:31 +00:00
Brian Gough
4bf90afe0e
update tests
2015-11-30 15:22:21 +00:00
Brian Gough
7a9577e081
only flush to track-changes when ops are queued
2015-11-30 15:17:11 +00:00
Brian Gough
529a061567
Merge pull request #4 from sharelatex/flush-ops-from-redis
...
push ops to track changes when document is flushed
2015-11-30 10:46:21 +00:00
James Allen
2589e2d417
Gracefully return when an op has already been submitted
...
It is not a fatal error if an op has already been submitted. We just
need to send an ack back to the client that submitted it and continue.
If we detect a duplicate op, set dup: true on the op and pass it back
to real-time for distributing. The dup: true flag will ensure it only
gets acknowledged to the submitting client, not everyone.
2015-11-19 10:54:28 +00:00
James Allen
e73890bfc2
Error if update would make document too long
2015-11-06 12:52:03 +00:00
James Allen
03e9d7390f
Add in mongo health check end point
2015-08-31 15:57:26 +01:00
James Allen
d27d2b90a2
Merge pull request #5 from sharelatex/utf16_fix
...
Replace UTF-16 surrogate characters with 'replacement character'
2015-06-12 10:18:34 +01:00
James Allen
333591d087
Extra null check
2015-06-12 10:16:33 +01:00