Brian Gough
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425
Merge branch 'master' into ho-flush-all-projects
2019-05-07 17:02:14 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Tim Alby
8b40da701e
update README
...
- remove build status badge
- change app name
- update copyright notice
- update links
2019-05-07 17:03:09 +02:00
Tim Alby
d316f172bf
update repo URL for Jenkins
2019-05-07 17:03:06 +02:00
Henry Oswald
e57741cb80
stub out project flusher for unit tests
2019-05-07 15:46:30 +01:00
Henry Oswald
4b8a27a220
change github url
2019-05-07 14:01:12 +01:00
Tim Alby
ca6bfd8929
Update README
2019-05-06 17:28:52 +02:00
Henry Oswald
daca83a057
add dryRun option to flush all projects
2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e
adds /flush_all_projects project
2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9
Merge pull request #48 from sharelatex/bg-check-incoming-hash
...
check incoming hash when present
2019-04-29 10:15:44 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
33478f95fd
Fix package.json versions
2019-04-11 16:32:31 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98
check version before it is modified by applyOp
2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b
check incoming hash when present
2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Henry Oswald
4d7f9f3c1a
add sentry into settings.defaults
2019-03-05 17:34:27 +00:00
Brian Gough
11f07d1f09
Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses
...
use explicit json content-type to avoid security issues with text/html
2019-02-22 09:24:30 +00:00
Henry Oswald
937e8fe071
use redis cluster key in defaults for pendingupdates
2019-02-15 13:58:29 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
0a0fc91f28
Update app.coffee
2019-02-12 13:27:13 +00:00
Henry Oswald
3bc4cb492a
added log line
2019-02-07 16:30:53 +00:00
Henry Oswald
08723f8972
revert health check redis types
2019-02-07 15:53:26 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
b5564095f3
add /health_check/redis route back in
2019-02-07 15:10:40 +00:00
Henry Oswald
73bd264401
remove realtime keys in settings, no longer used
2019-02-07 14:55:24 +00:00
Henry Oswald
2998750a33
fix redis version lock
2019-02-06 16:01:44 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Henry Oswald
8d3fb729c5
point to web_api
2019-02-01 20:10:47 +00:00
Henry Oswald
e44a02391a
standardise name of document-updater
2019-02-01 20:04:43 +00:00
Henry Oswald
98de529e42
make config look like chef config
2019-02-01 19:52:00 +00:00
Henry Oswald
3051d7db4b
bulk upgrade to 1.1.12, moved to npm rather than git
2019-01-31 16:00:24 +00:00
Henry Oswald
d279a9149c
add dispatcherCount to config file
2019-01-25 17:24:19 +00:00
Henry Oswald
76b054c7f4
bump node redis to 1.0.5
2019-01-25 17:01:59 +00:00
Christopher Hoskin
3eb2d27d4a
Add **/*.map to .gitignore
2019-01-08 15:01:48 +00:00
Christopher Hoskin
5db1913cec
Bump settings to v1.1.0
2019-01-08 15:00:26 +00:00
Christopher Hoskin
a783c6b3cc
Bump logger to 1.5.9
2019-01-04 09:33:08 +00:00
Christopher Hoskin
05a80c7cba
Add app.map to .gitignore
2019-01-04 09:31:00 +00:00
Christopher Hoskin
0aa7315c27
Bump buildscripts to 1.1.10
2019-01-04 09:24:09 +00:00
Christopher Hoskin
d9ec90f0e2
Move to metrics v2 and shrinkwrap
2019-01-04 09:22:09 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Henry Oswald
9437cf8b02
explicity split redis config values
2018-12-05 15:16:55 +00:00
Henry Oswald
5f046ed329
install metrics route and bump metrics version
2018-12-05 12:35:46 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Henry Oswald
a11d6dcc48
Merge branch 'master' into ho-docker
2018-10-15 15:17:40 +01:00
Alasdair Smith
c128e0ab1c
Use setting instead of hard-coding port
2018-10-09 11:46:27 +01:00
Henry Oswald
ff673c71d5
set redis port via env var
2018-09-28 17:04:34 +01:00
Henry Oswald
93cf574251
upgrade build scripts 1.1.9
2018-09-28 15:13:47 +01:00
Henry Oswald
3db7377d74
add process.env["REDIS_PASSWORD"]
2018-09-28 15:11:17 +01:00
Brian Gough
537bd9a0c0
Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web
...
limit resync requests to web (connects to overleaf/sharelatex#775 )
2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d
add more logging to resync operations
2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee
limit parallel resync doc requests to web
2018-08-16 11:13:11 +01:00
Brian Gough
fdcb806518
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb
Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue"
...
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
2018-07-23 16:05:04 +01:00
Brian Gough
33e898f671
update to 1.1.9 build scripts
2018-07-23 15:35:54 +01:00
Brian Gough
6237577f87
set a timestamp for the first entry in the projectHistory:Ops queue
2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2
move tk call to before sandboxed module call
2018-06-05 13:54:30 +01:00
Henry Oswald
581963a4af
Dockerised env. 1.1.3 build scripts
2018-05-23 13:52:20 +01:00
Brian Gough
7e526395fb
fix check for "delete component" message
2018-04-30 09:56:01 +01:00
Brian Gough
37f431f148
also block "delete component" messages from sentry
2018-04-27 16:03:46 +01:00
Brian Gough
64670f272a
stop OpRangeNotAvailable errors going to sentry
2018-04-27 15:45:28 +01:00
Hayden Faulds
af92ca70a1
coerce projectHistoryId to integer after reading from Redis
2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49
Revert "Revert "Add projectHistoryId to updates""
2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311
Revert "Add projectHistoryId to updates"
2018-04-23 11:41:29 +01:00
Hayden Faulds
9629d0a11e
Merge pull request #26 from sharelatex/hof-deleted-project-history
...
Add projectHistoryId to updates
2018-04-23 11:01:53 +01:00
Alberto Fernández Capel
7d597fe59e
Do not specify node version in .travis.yml
...
So travis picks instead the version from .nvmrc
2018-04-19 14:03:35 +01:00
Hayden Faulds
25c0be4eaa
add projectHistryId to project structure updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32
add projectHistoryId to resync updates
2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593
cache projectHistoryId with doc in Redis
2018-04-13 15:22:05 +01:00
Brian Gough
dd0f8b880a
change update to projectUpdate in project related methods
2018-03-16 10:54:12 +00:00
Brian Gough
779f00f912
add acceptance test for ordering of project structure changes
2018-03-15 16:27:14 +00:00
Brian Gough
3385d2640a
fix structure ordering bug
2018-03-15 16:27:10 +00:00
Brian Gough
75a5428cbf
update acceptance tests
2018-03-15 16:24:14 +00:00
Brian Gough
b3887fd984
update unit tests for incoming project versions
2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9
support project version on incoming requests
2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a
Merge pull request #21 from sharelatex/hof-history-sync
...
resync project history endpoint
2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3
extract ProjectHistoryRedisManager
2018-03-09 15:36:38 +00:00
Hayden Faulds
fc6ef6ea7a
improve logging
2018-03-09 12:21:48 +00:00
Hayden Faulds
088ab6e7ba
rename queueResyncDocContents -> queueResyncDocContent
2018-03-09 12:21:37 +00:00
Hayden Faulds
64d1fe13bc
additional logging for project history resync
2018-03-07 16:24:22 +00:00
Hayden Faulds
a72d228354
add unit tests
2018-03-07 13:29:53 +00:00
Hayden Faulds
152a25e96d
rename resyncProject -> resyncProjectHistory
2018-03-07 13:29:42 +00:00
Hayden Faulds
039f997e80
update project history resync url
2018-03-07 11:37:26 +00:00
Hayden Faulds
d46217be52
queue doc content sync updates
2018-03-02 15:02:27 +00:00
Hayden Faulds
52d6b710bd
add ability to resync project structure
2018-03-02 15:01:53 +00:00
James Allen
939eaa2d4b
Don't allow a document to be loaded without a pathname
2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3
remove unecessary method wrapping
2018-02-22 10:17:59 +00:00
Hayden Faulds
0f87ae1f74
simplify comments
2018-02-22 10:16:29 +00:00
Hayden Faulds
ea0dd9700b
reduce the number of times we flush project history
2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d
Merge pull request #18 from sharelatex/msw-flush-project-ops
...
Flush project structure changes every 500
2018-02-16 10:30:00 +00:00
James Allen
5425fdef8a
update to 1.0.1 build scripts
2018-02-15 16:54:21 +00:00
James Allen
657da70d45
Fix unit test namespacing from mocha upgrade
2018-02-15 16:28:58 +00:00
James Allen
5d1659457e
Update to build 1.0.0 build scripts
2018-02-15 16:28:40 +00:00
James Allen
f6c79f3203
Fix and extend unit tests
2018-02-15 15:55:12 +00:00
Michael Walker
6c57317f8d
Add a test that sending too few updates does not flush history
2018-01-31 11:41:08 +00:00
Michael Walker
241d1b27d5
Remove _shouldFlushHistoryOps wrapper
2018-01-31 11:27:40 +00:00
Michael Walker
772ee4083a
Turn a unit test into an acceptance test
2018-01-31 11:17:56 +00:00
Michael Walker
bdaa2ffa77
Add an acceptance test for flushing project changes
2018-01-26 14:41:47 +00:00
Michael Walker
d2865f0650
Correctly count project_ops_length
2018-01-26 14:41:28 +00:00
Michael Walker
3028fb9c3d
Only flush project updates when crossing the threshold
2018-01-24 11:57:03 +00:00
Michael Walker
24c74db0dc
Flush project changes when queue hits limit
2018-01-23 11:18:20 +00:00
Michael Walker
3d313ebc53
Add method to get number of queued project updates
2018-01-23 11:10:54 +00:00
James Allen
ccf6cb1a18
Provide hosts as environment settings, add npm run start script and update coffeescript
2017-12-29 08:13:16 +00:00
Brian Gough
0b583dd4e6
fix log message
2017-12-20 13:27:10 +00:00
Brian Gough
a8b14552c0
allow flush to succeed when doc is already deleted
2017-12-19 12:27:57 +00:00
Brian Gough
41f15c4fa8
fix unit test
2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf
Merge branch 'bg-limit-ops-in-lock'
2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d
Calculate and send doc length before each update
2017-12-11 17:26:32 +00:00
Hayden Faulds
bd45fcb5f8
Merge pull request #12 from sharelatex/hof-version-entity-creation
...
version entity creation; Connects to sharelatex/web-sharelatex-internal#104
2017-11-24 10:02:51 +00:00
Brian Gough
fb572670cd
Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects
...
use a separate parameter for doc and project flush
2017-11-21 13:54:03 +00:00
Brian Gough
0bb9f147aa
fix whitespace
2017-11-13 11:56:08 +00:00
Brian Gough
a4bbf5cf52
use a separate parameter for doc and project flush
2017-11-13 11:53:39 +00:00
Hayden Faulds
d0e56e5054
add acceptance tests for entity additions
2017-11-10 15:01:37 +00:00
Hayden Faulds
a3420b1236
version entity additions
2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac
add acceptance test for entity renaming
2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e
version file renames
2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23
version document renames
2017-11-08 09:20:37 +00:00
Hayden Faulds
b8052e7612
allow settings.redis.project_history to be undefined
2017-10-25 13:56:38 +01:00
Hayden Faulds
5b7be93c79
Merge pull request #2 from sharelatex/hof-doc-pathname
...
Add metadata to updates for history
2017-10-25 13:13:00 +01:00
Brian Gough
e590543e42
Merge pull request #4 from sharelatex/bg-fix-listen-in-acceptance-tests
...
exit if mock servers fail to start
2017-10-25 09:13:48 +01:00
Brian Gough
f3098f7470
upgrade ioredis to 3.2.1 via redis-sharelatex
2017-10-24 12:20:14 +01:00
Brian Gough
367ed1e941
Merge pull request #8 from sharelatex/bg-upgrade-to-node-6x
...
upgrade to node 6.9.5
2017-10-24 11:11:23 +01:00
Brian Gough
56fd6e3d48
upgrade to node 6.9.5
2017-10-24 11:08:20 +01:00
Brian Gough
d9d58393d7
remove unused ioredis package
...
it is loaded from redis-sharelatex, not here.
2017-10-24 10:39:00 +01:00
Hayden Faulds
d1f0c4ceae
add acceptance tests for project history API
2017-10-24 09:06:23 +01:00
Brian Gough
14f3d22071
upgrade to latest async ^2.5.0
2017-10-23 16:02:24 +01:00
Brian Gough
20655b6eeb
remove unused requires of async
2017-10-23 15:59:32 +01:00
Hayden Faulds
04ecd1e7ee
set pathname in acceptance tests
2017-10-23 15:32:25 +01:00
Hayden Faulds
be41a1614d
decorate ops with pathname and doc_length
2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8
flush track-changes and project-history in HistoryManager
2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c
conditionally enqueue history updates for project
2017-10-23 15:32:25 +01:00
Hayden Faulds
c27df0bfef
split apart multi and rclient in tests
2017-10-23 14:24:23 +01:00
Hayden Faulds
81f998afe9
return pathname from DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc
handle pathname in DocumentManager.getDoc
2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386
store pathname in Redis cache
2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a
return pathname from PersistenceManager
2017-10-23 14:17:14 +01:00
Brian Gough
02d3d1bd17
fix unit tests
...
timekeeper must be called before SandboxedModule.require
2017-10-23 14:02:53 +01:00
Brian Gough
7c2a28c161
fix use of timekeeper in unit tests
2017-10-23 13:33:14 +01:00
Brian Gough
4d82f1432a
Merge pull request #5 from sharelatex/bg-upgrade-ioredis-to-3x
...
update to ioredis 3
2017-10-23 13:27:02 +01:00
Brian Gough
448f131011
exit if mock servers fail to start
2017-10-20 14:56:12 +01:00
Joe Green
3df2d60948
Update Jenkinsfile
2017-10-16 14:14:06 +01:00
Joe Green
d387f979bc
only alert on master
2017-10-12 16:53:31 +01:00
Brian Gough
3954ecf85e
add missing timekeeper package
2017-10-12 11:23:24 +01:00
Brian Gough
08f0c67cbf
Merge pull request #3 from sharelatex/bg-auto-doc-flush
...
add unflushed time to doc in redis; connects to overleaf/sharelatex#190
2017-10-12 10:55:54 +01:00
Brian Gough
79c276ea77
update to ioredis 3
2017-10-11 17:00:01 +01:00
Brian Gough
727a534d01
keep the existing GET method temporarily
...
to avoid any problems due to deployment in the wrong order
2017-10-11 15:44:35 +01:00
Brian Gough
7cbb3e7af8
change getProjectDocs endpoint from GET to POST
...
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c
update DocumentManager.getDoc signature
...
keep alreadyLoaded flag at the end for consistency
2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005
add unflushed time to doc in redis
2017-10-06 12:34:28 +01:00
James Allen
340aa98de5
Add OSS syncing task
2017-10-04 11:27:19 +01:00
Brian Gough
ad745f133d
Merge pull request #59 from sharelatex/bg-reset-project-state
...
add endpoint to clear project state
2017-09-13 10:39:47 +01:00
Brian Gough
3842a27278
change .../clear endpoint to .../clearState
2017-09-12 11:39:18 +01:00
Brian Gough
2047bd5516
fix broken unit test
2017-09-11 09:44:05 +01:00
Brian Gough
f34c12fdc7
add endpoint to clear project state
2017-09-08 15:50:26 +01:00
Brian Gough
215a939d4e
add comment about log.error for excludeVersions
2017-09-08 13:43:22 +01:00
Brian Gough
1e54b4c72c
log all errors in getProjectDocs
2017-09-07 14:12:48 +01:00
Joe Green
c16c6c3bd3
added build.txt
2017-09-04 14:48:47 +01:00
Brian Gough
9d931a3fa7
fix broken unit tests
2017-08-18 12:08:39 +01:00
Brian Gough
230c93766d
fix broken RedisManager unit test
2017-08-18 11:59:31 +01:00
Brian Gough
8d408de875
Merge pull request #56 from sharelatex/bg-compile-from-redis
...
provide endpoint for current project docs in redis
2017-08-18 11:33:27 +01:00
Joe Green
0e8ce29414
compile acceptance tests
2017-08-14 15:33:30 +01:00
Joe Green
b1a7f779d1
add acceptance tests
2017-08-14 15:28:04 +01:00
Brian Gough
8372911a1b
return doclines as array from getProjectDocs
...
for compatibility with getDoc
2017-08-11 16:56:30 +01:00
Brian Gough
5971a19084
added acceptance tests
2017-08-11 16:55:31 +01:00
Brian Gough
f6be68eb08
added missing unit tests file
2017-08-11 16:49:16 +01:00
Joe Green
1146253c0a
Create Jenkinsfile
2017-08-11 11:06:36 +01:00
Brian Gough
f696ccb0d9
add unit tests for getProjectDocs
2017-08-10 16:41:28 +01:00
Brian Gough
1eb80936f2
fix log line
2017-08-10 14:57:27 +01:00
Brian Gough
47bebf9638
no need to support optional keys.projectState
2017-08-09 16:47:09 +01:00
Brian Gough
bd6b5b2c59
use v instead of rev in getProjectDocs
2017-08-09 15:30:53 +01:00
Brian Gough
ca7d9dce85
fix unit tests
2017-08-08 09:40:39 +01:00
Brian Gough
9f3ec72f81
switch to single get/set method for getProjectDocs
...
if project state hasn't changed, return the docs. Otherwise set the hash
and return a 409 Conflict response.
2017-08-07 14:43:28 +01:00
Brian Gough
f5f516a910
delete clsi state when deleting doc (if used)
2017-08-03 15:04:12 +01:00
Brian Gough
860537146b
avoid logging doclines in getProjectDocs
2017-08-03 15:04:12 +01:00
Brian Gough
80be5adc0d
change filter to exclude in getProjectDocs
2017-08-03 15:04:06 +01:00
Brian Gough
54e02cd895
update docupdater endpoint to /project/id/docs
2017-08-03 14:41:34 +01:00
Brian Gough
0109585092
provide endpoint for current project docs in redis
2017-08-02 13:42:10 +01:00
Shane Kilkelly
c3465cf3d4
Reduce retry delay to 10ms, and refactor
2017-07-18 15:28:18 +01:00
Shane Kilkelly
dc8d08c030
Update acceptance tests to reflect retry logic
2017-07-17 11:13:21 +01:00
Shane Kilkelly
0d46c3e2b3
WIP: auto-retry web requests
2017-07-17 10:08:21 +01:00
Brian Gough
c21cc013f3
use null instead of err in callbacks on success
2017-07-12 10:48:32 +01:00
Brian Gough
d2730c8d6b
unit tests for locking timeouts
2017-07-12 10:48:32 +01:00
Brian Gough
bb0dc4b4d5
release the lock if it took too long to acquire it
2017-07-12 10:48:32 +01:00
Brian Gough
54c0fc1180
put a 5 second timeout in getting redis lock
2017-07-11 14:28:58 +01:00
Brian Gough
ffea31290c
return total time from the profile .end() method
2017-07-11 14:28:58 +01:00
Brian Gough
41c2bd7fcd
Merge pull request #53 from sharelatex/bg-timeout-on-slow-prev-ops
...
add a timeout for getPreviousDocOps
2017-06-27 13:26:02 +01:00
Brian Gough
59ace9d5c5
avoid long lines in timeout check
2017-06-26 16:36:05 +01:00
Brian Gough
8e1e14c9ca
update to redis-sharelatex 1.0.3
...
adds keepalive support
2017-06-26 13:49:44 +01:00
Brian Gough
963e513057
add a timeout for getPreviousDocOps
...
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
2017-06-23 15:50:21 +01:00
James Allen
7b6966899b
Add script to expire existing DocOps lists
2017-06-14 12:09:34 +01:00
James Allen
54f128289f
Merge pull request #50 from sharelatex/bg-metric-for-unlock-errors
...
add metric for unlock errors
2017-06-13 14:46:40 +01:00
James Allen
7d22e4422c
Merge pull request #49 from sharelatex/bg-add-redis-timeout
...
add redis timeout
2017-06-13 14:46:32 +01:00
James Allen
30f57a7bfc
Merge pull request #51 from sharelatex/ja-expires-ordering
...
Ensure expires is called after the key is created
2017-06-13 14:05:22 +01:00
James Allen
770b53f6af
Add missing mocks to speed up tests and clean up output
2017-06-08 16:43:27 +01:00
James Allen
a3d7260619
Ensure expires is called after the key is created
2017-06-08 16:43:06 +01:00
Brian Gough
2e5d57cd99
don't increase rate limit when tasks are failing
2017-06-07 11:47:28 +01:00
Brian Gough
1ef258c878
add a timeout on the redis getDoc request
2017-06-07 11:34:42 +01:00
Brian Gough
673f4228ce
add metric for unlock errors
...
sentry does not record them reliably, due to rate limiting
2017-06-05 16:29:58 +01:00
Brian Gough
729216c9b1
add unit tests for DocsIn check
2017-05-31 16:08:45 +01:00
Brian Gough
12e8eaa9b6
fix bug in doclines check
2017-05-31 16:08:33 +01:00
Brian Gough
bed64d26c8
check if doc is missing from DocsIn set
2017-05-31 15:33:59 +01:00
Brian Gough
fa089dc28d
avoid graphite drift in active worker count
2017-05-30 14:37:01 +01:00
Brian Gough
891ffda3bf
put a rate limit on worker calls
2017-05-26 15:08:35 +01:00
Brian Gough
c43f97e8e3
add metric for active workers
2017-05-24 12:12:15 +01:00
Brian Gough
a3947c587f
fix bug, jsonOps is array not string
2017-05-22 15:20:28 +01:00
Brian Gough
54ad45c870
add null byte check for ranges
2017-05-22 15:00:05 +01:00
Brian Gough
a13f055d62
add null byte check for ops
2017-05-22 15:00:00 +01:00
Brian Gough
78828ebbc5
fine grained logging for cpu usage in applyUpdate
2017-05-19 16:00:16 +01:00
Brian Gough
a550078071
replace verbose logging by slow query log
2017-05-18 15:04:12 +01:00
Brian Gough
95596061d4
clarify calculation of process.hrtime
2017-05-18 15:02:54 +01:00
Brian Gough
e00f4dde8e
log profilers start and end times for reference
2017-05-18 15:02:44 +01:00
Brian Gough
e11f64f83a
increase profiling cutoff to 1s
2017-05-18 15:02:08 +01:00
Brian Gough
b6efb051e1
add profiling
2017-05-18 11:17:08 +01:00
Brian Gough
387ecac6cb
log for each get/try/release lock
2017-05-18 11:17:08 +01:00
Brian Gough
dff6e2c3da
bug fix in backoff
2017-05-16 16:31:28 +01:00
Brian Gough
c7d8fbbb8a
exponential backoff for lock up to maximum time
2017-05-16 15:22:02 +01:00
Brian Gough
b15d2ef796
Revert "fail safely if doc cannot be loaded"
2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4
Merge pull request #34 from sharelatex/pr-bulk-actions
...
Add methods to bulk accept changes.
2017-05-15 15:14:33 +01:00
Paulo Reis
ef7fa59257
Update ranges tracker.
2017-05-15 11:11:14 +01:00
Paulo Reis
05d7d1b8c6
Merge branch 'master' into pr-bulk-actions
2017-05-15 10:35:16 +01:00
Paulo Reis
36d16a76e0
Unit test accept changes in the ranges manager.
2017-05-12 14:42:40 +01:00
Brian Gough
2ee40d0748
ensure document is in redis before consuming ops
2017-05-12 14:22:18 +01:00
Brian Gough
be96548199
log number of updates for future debugging
2017-05-12 14:07:59 +01:00
Brian Gough
d4a8d88750
put a limit on the number of ops per iteration
2017-05-12 13:15:59 +01:00
Paulo Reis
d53266ea2d
Update web API stub.
2017-05-11 12:00:34 +01:00
Paulo Reis
5677c7ad9a
Update range tracker.
2017-05-11 11:31:19 +01:00
Paulo Reis
f690ef078a
Update tests; add new ones for multiple changes.
2017-05-09 16:16:25 +01:00
Brian Gough
36407ac726
rename HistoryManager pushUncompressedHistoryOps
2017-05-09 12:02:27 +01:00
Brian Gough
7ce6285e3d
increase flush threshold to 100 ops
2017-05-09 10:44:26 +01:00
Brian Gough
fdf5e8e0b8
get history ops length directly from redis update
2017-05-09 10:34:31 +01:00
Brian Gough
2d158b03d7
rename pushUncompressedHistoryOps
2017-05-09 09:32:56 +01:00
Paulo Reis
144804e516
Update tests with new methods.
2017-05-08 16:34:17 +01:00
Paulo Reis
d7ac57d054
Use new bulk method.
2017-05-08 16:08:42 +01:00
Paulo Reis
70341e4e80
Add bulk method from range tracker.
2017-05-08 16:08:34 +01:00
Brian Gough
e2f70aca1a
fix tests for redis cluster
2017-05-08 16:02:49 +01:00
Brian Gough
79d8fced49
make history update more atomic
2017-05-08 15:58:29 +01:00
Paulo Reis
e96e6d6a29
Remove unused method.
2017-05-05 15:17:19 +01:00
Paulo Reis
9b8a2c3030
Consolidate methods to accept changes.
2017-05-05 15:12:06 +01:00
James Allen
87a5e30a0b
Bump redis-sharelatex version
2017-05-05 13:37:58 +01:00
James Allen
7456238a71
Bump redis-sharelatex version
2017-05-04 15:42:10 +01:00
Paulo Reis
d530ee9501
Add methods to bulk accept changes.
2017-05-04 15:32:54 +01:00
James Allen
5f93640077
Add scripts for testing cluster failover scenarios
2017-05-04 11:14:17 +01:00
James Allen
4104ca4889
Add in separate redis config for the lock and fix a few web -> realtime
2017-05-03 16:27:32 +01:00
James Allen
5c3661d6d9
Merge remote-tracking branch 'origin/bg-redis-cluster-refactor' into ja-redis-cluster-refactor
2017-05-02 15:39:54 +01:00
James Allen
ed493d8ad3
Rename web -> realtime for consistency with realtime
2017-05-02 15:38:33 +01:00
Brian Gough
64aef0b55a
fix acceptance test to work with redis cluster too
2017-04-27 10:42:43 +01:00
Brian Gough
dc77bc207d
change acceptance test to use redis history client
2017-04-24 16:31:23 +01:00
James Allen
c5449ae282
Split out redis config for real-time and track-changes into separate cluster-compatible configs
2017-04-13 17:00:42 +01:00
James Allen
f21208e841
Use new redis-sharelatex instead of RedisBackend for cluster abstraction
2017-04-12 14:53:03 +01:00
Brian Gough
5583764d05
fix callback prototype in getLock
2017-03-30 15:31:47 +01:00
Brian Gough
c11618b475
improve unlock error handling
2017-03-30 15:31:34 +01:00
Brian Gough
b2b4bc44df
upgrade logger-sharelatex to v1.5.6
2017-03-30 14:16:35 +01:00
Brian Gough
527c02a00b
log op versions pushed to track changes
2017-03-30 11:20:50 +01:00
Shane Kilkelly
64658d0034
Add a .nvmrc file
2017-03-27 14:50:09 +01:00
James Allen
0245bfd031
Support an incoming undo flag for history restores
2017-03-21 11:20:38 +00:00
James Allen
a3a5524778
Fix broken unit test from merge
2017-03-20 10:37:02 +00:00
James Allen
199079a5b7
Merge branch 'master' into ja-cut-and-paste-comments
2017-03-20 10:29:01 +00:00
James Allen
9c5299ec7c
Update RangeTracker to support upserting and moving comments for cut and paste
2017-03-16 15:49:41 +00:00
James Allen
5499a67d78
Add in a consistency check after applying updates that ranges still match
2017-03-15 14:12:06 +00:00
James Allen
b9e4492001
Wrap RangeTracker in try catch
2017-03-06 19:05:11 +00:00
James Allen
a3a9a726d8
Update unit tests
2017-03-06 12:01:18 +00:00
James Allen
d8135749d2
Merge branch 'master' into ja-increase-limits-on-changes
...
Conflicts:
app/coffee/RedisManager.coffee
2017-03-06 11:57:40 +00:00
James Allen
30294dafb1
Merge pull request #23 from sharelatex/bg-upgrade-raven-logging
...
upgrade to logger-sharelatex 1.5.3
2017-03-06 11:53:47 +00:00
James Allen
269433f250
Merge pull request #25 from sharelatex/bg-log-doclines-on-hash-mismatch
...
log doclines on hash mismatch
2017-03-06 11:53:24 +00:00
Brian Gough
501d907299
upgrade to logger-sharelatex 1.5.4
2017-03-03 16:08:14 +00:00
Brian Gough
d086e0b61b
log doclines on hash mismatch
2017-03-03 15:57:44 +00:00
Brian Gough
3f13263ecf
upgrade to logger-sharelatex 1.5.3
2017-03-03 15:32:11 +00:00
Brian Gough
b186a01c04
don't log errors from redis backend
...
this also picks up errors from RedisManager like "doc ops range is not
loaded in redis"
2017-03-03 15:27:42 +00:00
James Allen
d56bb55953
Revert PR #19
2017-03-01 16:49:46 +00:00
James Allen
9cfe651930
Merge pull request #19 from sharelatex/ja-track-changes
...
Don't consume partial parts of delete when inserting a change
2017-03-01 10:20:06 +01:00
James Allen
f544814dda
Add in an absolute size limit on the ranges JSON object
2017-02-27 14:34:20 +01:00
James Allen
f211c282b2
Merge branch 'master' into ja-increase-limits-on-changes
2017-02-27 13:50:42 +01:00
James Allen
dd45a8dd7d
Improve logging around hashes and null byte errors
2017-02-27 10:06:24 +01:00
James Allen
60f3393c44
Do a naive check for null bytes after JSON.stringify on doc lines
2017-02-27 09:55:04 +01:00
James Allen
fd3dee0af5
Increase changes count limit to 2000
2017-02-24 14:58:47 +01:00
James Allen
80284e1b01
Only cancel deletes with inserts on undo and reject
2017-02-24 14:21:06 +01:00
James Allen
0420929e2a
Reduce the data logged when updating ranges
2017-02-23 13:57:47 +01:00
James Allen
e1481df76a
Don't consume partial parts of delete when inserting a change
2017-02-23 11:39:19 +01:00
Brian Gough
49271d4ee4
update logger to 1.5.2 - rate limiting on sentry
2017-02-23 10:15:56 +00:00
James Allen
5ab903169a
evert "update logger to 1.5.2 - rate limiting on sentry"
...
This reverts commit ba1c4a84959cd2eb33d7d624239d3e621ebae1fd.
2017-02-22 18:26:42 +01:00
James Allen
d6b5cc6389
Increase changes limit to 1200
2017-02-22 18:11:55 +01:00
Brian Gough
ed59ca32fd
update logger to 1.5.2 - rate limiting on sentry
2017-02-22 15:58:40 +00:00
Brian Gough
168bab6960
make hash error logging fine grained
2017-02-22 14:32:51 +00:00
Brian Gough
19e7522d3f
make logging of hash errors optional
2017-02-22 13:46:34 +00:00
Brian Gough
8d1b11f37a
Merge branch 'bg-verify-writes-II'
2017-02-21 16:03:48 +00:00
Brian Gough
5eb339e525
fix utf8 encoding in sha1 hash
2017-02-21 16:03:06 +00:00
Brian Gough
6b808522dc
Merge pull request #18 from sharelatex/bg-verify-writes
...
merge redis hash check
2017-02-20 15:55:19 +00:00
Brian Gough
590f8e7ced
Merge pull request #15 from sharelatex/bg-verify-writes
...
store sha1 hash of docLines in redis
2017-02-20 15:53:56 +00:00
Brian Gough
62165ddeab
add unit tests
2017-02-20 15:33:19 +00:00
Brian Gough
c57d3ce31c
compute hash on write in redis server
2017-02-20 14:09:09 +00:00
Brian Gough
a3c127e469
added unit tests
2017-02-20 13:53:25 +00:00
Shane Kilkelly
37172a2115
Debug Acceptance Tests: Try without npm rebuild
2017-02-20 09:22:54 +00:00