Skip to contents

Changelog

Source: NEWS.md

Version 4.4.0

  • Date 2024-11-08
  • When querying with only_attributes=F in binary or arrow format via the httpapi, uses the arrow+ and binary+ formats in both the use_aio=F and use_aio=T cases, and never instead modifies the AFL by adding in the flatten operator.
  • When querying with use_aio=T in text formats via the httpapi, actually uses aio_save instead of save. Adds a scidb.aio_text R option that switches the default value of use_aio flag from F to T for text formats. The behavior of the existing scidb.aio R options is unchanged; as before it switches the default value of use_aio from F to T for the binary format. The arrow format always uses aio_save irrespective of use_aio.
  • The above are achieved by using new support in the aio_save operator for arrow+ and binary+ format types and for text format strings compatible with those used by save. Thus, this version requires extra-scidb-libs-23.10 to be updated to package release 5-1 or greater on the SciDB server.
  • Adds test coverage for binary and text format queries using AIO by testing many existing unit test cases over a matrix of formats and use_aio settings.
  • No changes made to queries via shim

Version 4.3.0

  • Date 2024-10-15
  • error$message is now a single string with newlines instead of a vector (#353)
  • Results returned in arrow format are now returned as data.frame instead of tibble (#341)
  • Tests are now run on R 4.2.3, 4.3.3, and 4.4.1; this is now configured in the workflow file itself instead of using repo vars. (#358, #360)

Version 4.2.0

  • Date 2024-07-12
  • Bug fixes for parallel uploads in the SciDB HTTP API (SDB-8488, SDB-8509): If the SciDB server returns HTTP status code 429 “Too Many Requests”, the client backs off and retries its request after waiting the number of seconds indicated in the response’s “Retry-After” header.
  • Bug fix for “$ operator invalid” message during upload error (SDB-8524).

Version 4.1.3

  • Date 2024-06-13
  • Limit evaluation of R function calls in expression parsing to “c”, “paste0”, and boolean functions and predicates, in order to fix compatibility issues and allow the eventual replacement of ArrayOpR expression parsing with the expression parsing in this base package.

Version 4.1.2

  • Date: 2024-05-02
  • Work around SDB-8481 by setting default inactivity_timeout to NULL for SciDB HTTP Client API. The inactivity_timeout is now configurable using options(scidb.httpapi.inactivity_timeout).

Version 4.1.1

  • Date: 2024-03-24
  • Belated news for verison 4.1: version 4.1 included support for parsing R expressions bound to new variable names via scidb::assign_expressions(...) and scidb::assign_expressions_to_afl(...), as well as adding support to parse if (cond) {a} else {b}, if (cond) a else b, and ifelse(cond, a, b) expressions and map them to the AFL iif function.
  • Corrections to the github workflow for generating documentation, which prevented the automatic deployment of documentation for version 4.1

Version 4.1

  • Date: 2024-01-15

  • When using the SciDB HTTP API against a SciDB server of version 23.10.3 or later, clients can set options(scidb.httpapi.resumable=TRUE) to opt-in to using the experimental “timeout and resume” feature for queries. This lets the scidb server detect when the client is actively reading from a query, allowing it to extend the lifetimes of the query and authentication token so they don’t expire while the SciDBR application is running.

    Querying functions (like iquery()) take new optional arguments:
    • progress_check_secs: after every interval of this many seconds, interrupt the query, call progress_check_callback if provided, and resume the query.
    • progress_check_callback: a callback function which is called every progress_check_secs seconds, of the form: function(query, page_number, response_number, cumulative_nbytes, ...) The callback function can do things like displaying the number of bytes received so far. If the function stops execution using stop(), the query gets canceled.

    You can set options(scidb.httpapi.progress_check_secs) to control the default for progress_check_secs.

Version 4.0.5

  • Date: 2024-01-15
  • Bugfixes for failures uploading data frames via the SCIDB HTTP API found in the ArrayOpR unit tests

Version 4.0.4

  • Date: 2023-11-07
  • Speed up large fetches in Arrow format over SciDB HTTP API

Version 4.0.3

  • Date: 2023-09-28
  • Changes to CI configuration

Version 4.0.2

  • Date: 2023-07-24
  • Fix spurious warnings during garbage collection

Version 4.0.1

  • Date: 2023-07-19
  • Dependency fixes: R6 added to Imports; bit64 moved from Depends to Imports; glue::glue() usage replaced with paste()/paste0()

Version 4.0.0

  • Date: 2023-05-05
  • SciDBR can now connect directly to SciDB versions 23.2 and higher using the new SciDB HTTP API, without requiring Shim. To use it, connect using scidbconnect() as normal, but specify the HTTP API port number (usually 7000 + the SciDB port number; most often 8239). SciDBR will automatically determine the protocol and API to use, using the SciDB HTTP API for direct connections or the Shim API for shim connections.

Version 3.1.6

Version 3.1.5

  • Date: 2023-04-10
  • Fix bug – passing curl options as a list was not working in R>=4.2 (fix #279)

Version 3.1.4

  • Date: 2023-02-06
  • Fix bug – when using sort but no AIO, SciDBR may produce an error when using sort() (fix #263)

Version 3.1.3

  • Date: 2023-01-04
  • Consolidate curl handle creation code and add ability to set this via package config (fix #253)

Version 3.1.2

Version 3.1.1

Version 3.1.0

Version 3.0.10

  • Date: 2021-11-17
  • Fix for datetimes returned in situations where the client and server are not in the same time zone

Version 3.0.9

  • Date: 2021-10-18
  • Fix for #236 (scidbr objects are not printing to console)

Version 3.0.8

Version 3.0.7

  • Date: 2021-09-06
  • In as.scidb, added a new optional parameter - max_byte_size (default value: 500MB) for uploading R vectors in a multi-part fashion into SciDB. Example:
scidb::as.scidb(db, x = c(1:10), gc = TRUE)
scidb::as.scidb(db, x = c(1:10)/10, max_byte_size = 8, gc = TRUE)

Version 3.0.6

  • Date: 2021-08-13
  • In as.R, SciDB data frames are downloaded without the artificial dimensions

Version 3.0.5

Version 3.0.4

  • Date: 2021-05-26
  • Fix for uploading character vectors containing NA values

Version 3.0.3

Version 2.0.0

CRAN release: 2017-04-14

  • This is a major release that breaks API compatibility with previous package releases. Array objects have been removed. All SciDB arrays are now presented as virtual data frames in R. This change was informed by the most common uses we’ve seen.
  • Greatly simplified the package, improved data transfer performance
  • Support for SciDB 16.9 and older versions at least back to 15.7

Version 1.1-3

  • Support for upcoming SciDB 14.12, untested backwards support for older releases

  • Improved type parsing support, new binary transfer option for data frame objecs. This is much faster than UTF-8 transfer from SciDB.

  • Support for streaming data from SciDB.

  • Option to disable interrupt handling for faster large data transfers.

  • Added functions include order, more flexible dist, peek.

Version 1.1-2

CRAN release: 2014-04-16

SCIDB SUPPORT

  • Support for SciDB 14.3, untested support for older SciDB releases

  • Dropping arrays from other R sessions requires a non-default option facilitating multiple-user settings.

  • Queries can be canceled with R user interrupts now (for example with CTRL + C or ESC).

IMPROVEMENTS

  • Many improvements to merge and aggregate functions

  • N-d arrays with multiple attributes can now index with $

  • Improved R save/load of R data files containing SciDB objects

NEW FEATURES

  • Support for Paradigm4 glm and a simple glm model matrix builder

  • Support for Paradigm4 truncated SVD routine

  • Added hist, quantile, all.equal, antijoin, a c (SciDB concat-like) function and many others

Version 1.1-1

CRAN release: 2014-01-25

WINDOWED AGGREGATES

  • The aggregate function now supports moving and fixed window aggregates.

TLS/SSL ENCRYPTION AND AUTHENTICATION

  • The package now supports TLS/SSL encrypted communication with the shim network service and authentication. The only function affected by this is scidbconnect – simply supply an SSL port number and username and password arguments.

LAZY EVALUATION

  • Most scidb and scidbdf object can now represent array promises, un-evaluated SciDB queries equipped with a result schema and a context environment.

NEW COMPOSABLE OPERATORS

  • Many new functions were introduced that closely follow underlying SciDB AFL operators. All of the new functions are composable with lazy evaluation of the underlying query using the new scidbexpr class. Results are only computed and stored by SciDB when required or explicitly requested.

R SPARSE MATRIX SUPPORT

  • The package now maps sparse SciDB arrays to sparse R matrices. Only matrices (2-d arrays) with double-precision attributes are supported. Sparse array arithmetic uses the new P4 sparse matrix multiply operator when available.

USING RCurl NOW

  • We introduced a dependency on RCurl in order to support SSL and authentication with the SciDB shim service.

NO MORE SciDB NID

  • The package no longer tries to support SciDB arrays with NID dimensions, which never really worked anyway. Instead, many functions now take advantage of the new SciDB uniq and index_lookup operators if available (>=SciDB 13.6). Future package versions will take this further and introduce array dimension labeling using the new operators.

MANY NEW DATABASE-LIKE FEATURES

  • Aggregate was improved, merge sort, unique, index_lookup, and other new functions were added. See the vignette for more information.

Version 1.1-0

CRAN release: 2013-04-13

SIGNIFICANT BUG FIX:

  • Materializing subsetting operations could return inconsistently ordered data when results spanned SciDB array chunks across multiple SciDB instances. Data are now returned correctly in such cases.

OTHER BUG FIXES:

  • Fixed a bug in the processing of the start argument in the as.scidb function.

  • Fixed several bugs in the image function.

NEW FEATURES:

  • The iquery function now accepts n=Inf to efficiently download all output from query at once. The iterative=TRUE option should still be used with smaller n values to iterate over large results.

  • The crossprod and tcrossprod functions are now available for SciDB arrays and mixed SciDB/R objects.

  • A diag function is now available for SciDB matrices, returning result as a new SciDB 1-D array (vector).

  • Element-wise exponentiation was implemented for scidb array objects.

  • Implemented a sum function for scidb array objects.