Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Small improvements to object store #52

Open
wants to merge 7 commits into
base: object-store-0.10.2
Choose a base branch
from
Open

Small improvements to object store #52

Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8fb7cb2
small improvements
fsdvh Sep 26, 2024
3c49dd6
empty
fsdvh Sep 26, 2024
d174679
atomic value part_idx
fsdvh Sep 27, 2024
60db4c6
revert some changes
fsdvh Sep 27, 2024
09bcd4a
update
fsdvh Sep 27, 2024
ef03546
change finish method
fsdvh Oct 21, 2024
545fd20
Add checksum (#61)
avantgardnerio Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions object_store/src/aws/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ use futures::stream::BoxStream;
use futures::{StreamExt, TryStreamExt};
use reqwest::header::{HeaderName, IF_MATCH, IF_NONE_MATCH};
use reqwest::{Method, StatusCode};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::{sync::Arc, time::Duration};
use url::Url;

Expand Down Expand Up @@ -215,7 +216,7 @@ impl ObjectStore for AmazonS3 {
let upload_id = self.client.create_multipart(location, opts).await?;

Ok(Box::new(S3MultiPartUpload {
part_idx: 0,
part_idx: AtomicUsize::new(0),
state: Arc::new(UploadState {
client: Arc::clone(&self.client),
location: location.clone(),
Expand Down Expand Up @@ -319,7 +320,7 @@ impl ObjectStore for AmazonS3 {

#[derive(Debug)]
struct S3MultiPartUpload {
part_idx: usize,
fsdvh marked this conversation as resolved.
Show resolved Hide resolved
part_idx: AtomicUsize,
state: Arc<UploadState>,
}

Expand All @@ -334,8 +335,7 @@ struct UploadState {
#[async_trait]
impl MultipartUpload for S3MultiPartUpload {
fn put_part(&mut self, data: PutPayload) -> UploadPart {
let idx = self.part_idx;
self.part_idx += 1;
let idx = self.part_idx.fetch_add(1, Ordering::AcqRel);
let state = Arc::clone(&self.state);
Box::pin(async move {
let part = state
Expand All @@ -348,7 +348,8 @@ impl MultipartUpload for S3MultiPartUpload {
}

async fn complete(&mut self) -> Result<PutResult> {
let parts = self.state.parts.finish(self.part_idx)?;
let part_idx = self.part_idx.load(Ordering::Acquire);
let parts = self.state.parts.finish(part_idx)?;

self.state
.client
Expand Down
Loading